2026-03-13 00:51:01.520114 | Job console starting... 2026-03-13 00:51:01.529398 | Updating repositories 2026-03-13 00:51:01.550871 | Preparing job workspace 2026-03-13 00:51:07.288937 | Running Ansible setup... 2026-03-13 00:51:11.242482 | PRE-RUN START: [trusted : review.rdoproject.org/config/playbooks/base-minimal/pre.yaml@master] 2026-03-13 00:51:11.894406 | 2026-03-13 00:51:11.894535 | PLAY [localhost] 2026-03-13 00:51:11.904383 | 2026-03-13 00:51:11.904512 | TASK [Gathering Facts] 2026-03-13 00:51:12.939917 | localhost | ok 2026-03-13 00:51:12.959753 | 2026-03-13 00:51:12.959930 | TASK [log-inventory : Ensure Zuul Ansible directory exists] 2026-03-13 00:51:13.422947 | localhost -> localhost | changed 2026-03-13 00:51:13.430147 | 2026-03-13 00:51:13.430270 | TASK [log-inventory : Copy ansible inventory to logs dir] 2026-03-13 00:51:14.301922 | localhost -> localhost | changed 2026-03-13 00:51:14.321770 | 2026-03-13 00:51:14.321942 | TASK [Setup log path fact] 2026-03-13 00:51:14.348274 | localhost | ok 2026-03-13 00:51:14.373409 | 2026-03-13 00:51:14.373539 | TASK [set-zuul-log-path-fact : Set log path for a build] 2026-03-13 00:51:14.403893 | localhost | ok 2026-03-13 00:51:14.415214 | 2026-03-13 00:51:14.415324 | TASK [emit-job-header : Print job information] 2026-03-13 00:51:14.455852 | # Job Information 2026-03-13 00:51:14.456048 | Ansible Version: 2.15.12 2026-03-13 00:51:14.456086 | Job: vexxhost-hotstack-3-nodes 2026-03-13 00:51:14.456115 | Pipeline: github-experimental-trigger 2026-03-13 00:51:14.456141 | Executor: ze02.softwarefactory-project.io 2026-03-13 00:51:14.456167 | Triggered by: https://github.com/openstack-k8s-operators/hotstack/pull/235 2026-03-13 00:51:14.456194 | Log URL (when completed): https://logserver.rdoproject.org/46a/rdoproject.org/46a5d1853f074c73b1d21511708ab398/ 2026-03-13 00:51:14.456221 | Event ID: 8cf9a060-1e76-11f1-9a2b-3183ea4da38e 2026-03-13 00:51:14.461601 | 2026-03-13 00:51:14.461728 | LOOP [emit-job-header : Print node information] 2026-03-13 00:51:14.574713 | localhost | ok: 2026-03-13 00:51:14.575043 | localhost | # Node Information 2026-03-13 00:51:14.575083 | localhost | Inventory Hostname: hotstack 2026-03-13 00:51:14.575114 | localhost | Hostname: np0005645867 2026-03-13 00:51:14.575141 | localhost | Username: zuul 2026-03-13 00:51:14.575168 | localhost | Distro: CentOS 9 2026-03-13 00:51:14.575192 | localhost | Provider: vexxhost-nodepool-tripleo 2026-03-13 00:51:14.575215 | localhost | Region: RegionOne 2026-03-13 00:51:14.575237 | localhost | Label: cloud-centos-9-stream-tripleo-vexxhost 2026-03-13 00:51:14.575262 | localhost | Product Name: OpenStack Nova 2026-03-13 00:51:14.575289 | localhost | Interface IP: 38.102.83.155 2026-03-13 00:51:14.607419 | 2026-03-13 00:51:14.607544 | PLAY [all] 2026-03-13 00:51:14.617508 | 2026-03-13 00:51:14.617608 | TASK [Gather network facts] 2026-03-13 00:51:15.031146 | hotstack | ok 2026-03-13 00:51:15.062993 | 2026-03-13 00:51:15.063168 | TASK [include_role : start-zuul-console] 2026-03-13 00:51:15.085001 | hotstack | ok 2026-03-13 00:51:15.103075 | 2026-03-13 00:51:15.103193 | TASK [start-zuul-console : Start zuul_console daemon.] 2026-03-13 00:51:15.465556 | hotstack | ok 2026-03-13 00:51:15.485556 | 2026-03-13 00:51:15.485721 | TASK [include_role : add-build-sshkey] 2026-03-13 00:51:15.523034 | hotstack | ok 2026-03-13 00:51:15.552066 | 2026-03-13 00:51:15.552204 | TASK [add-build-sshkey : Check to see if ssh key was already created for this build] 2026-03-13 00:51:15.811554 | hotstack -> localhost | ok 2026-03-13 00:51:15.819035 | 2026-03-13 00:51:15.819163 | TASK [add-build-sshkey : Create a new key in workspace based on build UUID] 2026-03-13 00:51:15.848233 | hotstack | ok 2026-03-13 00:51:15.862864 | hotstack | included: /var/lib/zuul/builds/46a5d1853f074c73b1d21511708ab398/trusted/project_1/opendev.org/zuul/zuul-jobs/roles/add-build-sshkey/tasks/create-key-and-replace.yaml 2026-03-13 00:51:15.869071 | 2026-03-13 00:51:15.869146 | TASK [add-build-sshkey : Create Temp SSH key] 2026-03-13 00:51:16.485086 | hotstack -> localhost | Generating public/private rsa key pair. 2026-03-13 00:51:16.485483 | hotstack -> localhost | Your identification has been saved in /var/lib/zuul/builds/46a5d1853f074c73b1d21511708ab398/work/46a5d1853f074c73b1d21511708ab398_id_rsa. 2026-03-13 00:51:16.485569 | hotstack -> localhost | Your public key has been saved in /var/lib/zuul/builds/46a5d1853f074c73b1d21511708ab398/work/46a5d1853f074c73b1d21511708ab398_id_rsa.pub. 2026-03-13 00:51:16.485626 | hotstack -> localhost | The key fingerprint is: 2026-03-13 00:51:16.485714 | hotstack -> localhost | SHA256:L/yPBSN1Ed5wUDI7tTgWZW/KLj75DMdXKcRjyvxS/u8 zuul-build-sshkey 2026-03-13 00:51:16.485768 | hotstack -> localhost | The key's randomart image is: 2026-03-13 00:51:16.485818 | hotstack -> localhost | +---[RSA 3072]----+ 2026-03-13 00:51:16.485867 | hotstack -> localhost | | O=* | 2026-03-13 00:51:16.485917 | hotstack -> localhost | | ..& o | 2026-03-13 00:51:16.485963 | hotstack -> localhost | | . O=o o| 2026-03-13 00:51:16.486013 | hotstack -> localhost | | .oo++.o.| 2026-03-13 00:51:16.486059 | hotstack -> localhost | | S o+ oo..| 2026-03-13 00:51:16.486104 | hotstack -> localhost | | . o o+o. .| 2026-03-13 00:51:16.486179 | hotstack -> localhost | | o ..+++ .| 2026-03-13 00:51:16.486232 | hotstack -> localhost | | o ++=.. | 2026-03-13 00:51:16.486303 | hotstack -> localhost | | o.ooooE| 2026-03-13 00:51:16.486349 | hotstack -> localhost | +----[SHA256]-----+ 2026-03-13 00:51:16.486449 | hotstack -> localhost | ok: Runtime: 0:00:00.190986 2026-03-13 00:51:16.502029 | 2026-03-13 00:51:16.502201 | TASK [add-build-sshkey : Remote setup ssh keys (linux)] 2026-03-13 00:51:16.526803 | hotstack | ok 2026-03-13 00:51:16.541785 | hotstack | included: /var/lib/zuul/builds/46a5d1853f074c73b1d21511708ab398/trusted/project_1/opendev.org/zuul/zuul-jobs/roles/add-build-sshkey/tasks/remote-linux.yaml 2026-03-13 00:51:16.554450 | 2026-03-13 00:51:16.554558 | TASK [add-build-sshkey : Remove previously added zuul-build-sshkey] 2026-03-13 00:51:16.579588 | hotstack | skipping: Conditional result was False 2026-03-13 00:51:16.587842 | 2026-03-13 00:51:16.587933 | TASK [add-build-sshkey : Enable access via build key on all nodes] 2026-03-13 00:51:17.050263 | hotstack | changed 2026-03-13 00:51:17.058967 | 2026-03-13 00:51:17.059099 | TASK [add-build-sshkey : Make sure user has a .ssh] 2026-03-13 00:51:17.336528 | hotstack | ok 2026-03-13 00:51:17.352250 | 2026-03-13 00:51:17.352394 | TASK [add-build-sshkey : Install build private key as SSH key on all nodes] 2026-03-13 00:51:18.048142 | hotstack | changed 2026-03-13 00:51:18.062000 | 2026-03-13 00:51:18.062212 | TASK [add-build-sshkey : Install build public key as SSH key on all nodes] 2026-03-13 00:51:18.754518 | hotstack | changed 2026-03-13 00:51:18.767199 | 2026-03-13 00:51:18.767339 | TASK [add-build-sshkey : Remote setup ssh keys (windows)] 2026-03-13 00:51:18.792172 | hotstack | skipping: Conditional result was False 2026-03-13 00:51:18.799869 | 2026-03-13 00:51:18.799965 | TASK [remove-zuul-sshkey : Remove master key from local agent] 2026-03-13 00:51:19.201931 | hotstack -> localhost | changed 2026-03-13 00:51:19.216408 | 2026-03-13 00:51:19.216546 | TASK [add-build-sshkey : Add back temp key] 2026-03-13 00:51:19.495215 | hotstack -> localhost | Identity added: /var/lib/zuul/builds/46a5d1853f074c73b1d21511708ab398/work/46a5d1853f074c73b1d21511708ab398_id_rsa (zuul-build-sshkey) 2026-03-13 00:51:19.495394 | hotstack -> localhost | ok: Runtime: 0:00:00.014015 2026-03-13 00:51:19.501153 | 2026-03-13 00:51:19.501217 | TASK [add-build-sshkey : Verify we can still SSH to all nodes] 2026-03-13 00:51:19.836790 | hotstack | ok 2026-03-13 00:51:19.844472 | 2026-03-13 00:51:19.844562 | TASK [add-build-sshkey : Verify we can still SSH to all nodes (windows)] 2026-03-13 00:51:19.869393 | hotstack | skipping: Conditional result was False 2026-03-13 00:51:19.884717 | 2026-03-13 00:51:19.884819 | TASK [include_role : validate-host] 2026-03-13 00:51:19.906090 | hotstack | ok 2026-03-13 00:51:19.951431 | 2026-03-13 00:51:19.951542 | TASK [validate-host : Define zuul_info_dir fact] 2026-03-13 00:51:19.980726 | hotstack | ok 2026-03-13 00:51:19.985658 | 2026-03-13 00:51:19.985748 | TASK [validate-host : Ensure Zuul Ansible directory exists] 2026-03-13 00:51:20.214037 | hotstack -> localhost | ok 2026-03-13 00:51:20.226884 | 2026-03-13 00:51:20.227039 | TASK [validate-host : Collect information about the host] 2026-03-13 00:51:20.939286 | hotstack | ok 2026-03-13 00:51:20.952703 | 2026-03-13 00:51:20.952817 | TASK [validate-host : Sanitize hostname] 2026-03-13 00:51:21.014383 | hotstack | ok 2026-03-13 00:51:21.021280 | 2026-03-13 00:51:21.021361 | TASK [validate-host : Write out all ansible variables/facts known for each host] 2026-03-13 00:51:21.526450 | hotstack -> localhost | changed 2026-03-13 00:51:21.532129 | 2026-03-13 00:51:21.532203 | TASK [validate-host : Collect information about zuul worker] 2026-03-13 00:51:21.879650 | hotstack | ok 2026-03-13 00:51:21.887244 | 2026-03-13 00:51:21.887357 | TASK [validate-host : Write out all zuul information for each host] 2026-03-13 00:51:22.332852 | hotstack -> localhost | changed 2026-03-13 00:51:22.344881 | 2026-03-13 00:51:22.345010 | TASK [include_role : prepare-workspace-openshift] 2026-03-13 00:51:22.358765 | hotstack | skipping: Conditional result was False 2026-03-13 00:51:22.365744 | 2026-03-13 00:51:22.365881 | TASK [include_role : remove-zuul-sshkey] 2026-03-13 00:51:22.380878 | hotstack | skipping: Conditional result was False 2026-03-13 00:51:22.389552 | 2026-03-13 00:51:22.389750 | LOOP [ensure-output-dirs : Empty Zuul Output directories by removing them] 2026-03-13 00:51:22.647285 | hotstack | ok: "logs" 2026-03-13 00:51:22.647500 | hotstack | ok: All items complete 2026-03-13 00:51:22.647528 | 2026-03-13 00:51:22.878302 | hotstack | ok: "artifacts" 2026-03-13 00:51:23.074865 | hotstack | ok: "docs" 2026-03-13 00:51:23.091010 | 2026-03-13 00:51:23.091117 | LOOP [ensure-output-dirs : Ensure Zuul Output directories exist] 2026-03-13 00:51:23.324460 | hotstack | changed: "logs" 2026-03-13 00:51:23.538749 | hotstack | changed: "artifacts" 2026-03-13 00:51:23.723428 | hotstack | changed: "docs" 2026-03-13 00:51:23.750496 | 2026-03-13 00:51:23.750580 | PLAY RECAP 2026-03-13 00:51:23.750624 | hotstack | ok: 22 changed: 9 unreachable: 0 failed: 0 skipped: 5 rescued: 0 ignored: 0 2026-03-13 00:51:23.750649 | localhost | ok: 6 changed: 2 unreachable: 0 failed: 0 skipped: 0 rescued: 0 ignored: 0 2026-03-13 00:51:23.750693 | 2026-03-13 00:51:23.857138 | PRE-RUN END RESULT_NORMAL: [trusted : review.rdoproject.org/config/playbooks/base-minimal/pre.yaml@master] 2026-03-13 00:51:23.859264 | PRE-RUN START: [trusted : review.rdoproject.org/config/playbooks/hotstack/pre_stage1.yaml@master] 2026-03-13 00:51:24.474238 | 2026-03-13 00:51:24.474360 | PLAY [Create necessary files to deploy hotstack scenario] 2026-03-13 00:51:24.495499 | 2026-03-13 00:51:24.495602 | TASK [prepare-workspace : Start zuul_console daemon.] 2026-03-13 00:51:24.915051 | hotstack | ok 2026-03-13 00:51:24.931584 | 2026-03-13 00:51:24.931867 | TASK [prepare-workspace : Synchronize src repos to workspace directory.] 2026-03-13 00:51:27.548226 | hotstack | Output suppressed because no_log was given 2026-03-13 00:51:27.561100 | 2026-03-13 00:51:27.561190 | LOOP [add-authorized-keys : Enable access via build key on all nodes] 2026-03-13 00:51:28.000696 | hotstack | changed: 2026-03-13 00:51:28.000888 | hotstack | { 2026-03-13 00:51:28.000924 | hotstack | "public_key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCINPOs6XsfMVQMqjmiduT3Bxhogy7MbAFOWlR+9LlKkTW1kV6glsCXY5svtggd2zRtc40UWalHaj9jC+yEkZdI35h2PkK+Sww52u4nczqgNpIFNt7Dm2xtYKbTolAV9jSeNkRlGMCq9EBzQlwiV0HkHx1CdNR/zzXbyNZLmOybFkXqR150H01kUhdv0v6AMWTDZjMEWO/PaWCOgzZhH9ztXjy+rwa7pdaI4+0Pb9Kib+d8/IrgNsLB/sCazqD+9/COI6AHqzTDS1aU9/gMQrUN1ronzMEQhzzOuoI8WeLqU++j+0dT2UWVPlQqoln+RYu7dUn/DSZDrsYenudT5+P" 2026-03-13 00:51:28.000952 | hotstack | } 2026-03-13 00:51:28.029811 | 2026-03-13 00:51:28.029973 | TASK [Assert cloud name and cloud secret] 2026-03-13 00:51:28.089107 | hotstack | Output suppressed because no_log was given 2026-03-13 00:51:28.101178 | 2026-03-13 00:51:28.101343 | TASK [Ensure openstack release repo is installed] 2026-03-13 00:51:32.295433 | hotstack | changed 2026-03-13 00:51:32.313633 | 2026-03-13 00:51:32.313872 | TASK [Install openstack client packages] 2026-03-13 00:51:48.699564 | hotstack | changed 2026-03-13 00:51:48.711360 | 2026-03-13 00:51:48.711491 | TASK [Ensure HotStack work dir] 2026-03-13 00:51:49.031238 | hotstack | changed 2026-03-13 00:51:49.040594 | 2026-03-13 00:51:49.040752 | LOOP [Ensure etc directories] 2026-03-13 00:51:49.293345 | hotstack | changed: "/etc/openstack" 2026-03-13 00:51:49.293618 | hotstack | changed: All items complete 2026-03-13 00:51:49.293656 | 2026-03-13 00:51:49.568545 | hotstack | changed: "/etc/hotstack" 2026-03-13 00:51:49.588907 | 2026-03-13 00:51:49.589080 | TASK [Ensure .ssh dir exists] 2026-03-13 00:51:49.823950 | hotstack | Output suppressed because no_log was given 2026-03-13 00:51:49.830048 | 2026-03-13 00:51:49.830139 | TASK [Ensure ssh keypair exists] 2026-03-13 00:52:00.379026 | hotstack | Output suppressed because no_log was given 2026-03-13 00:52:00.388781 | 2026-03-13 00:52:00.388869 | TASK [Generate CA cert file] 2026-03-13 00:52:00.413528 | hotstack | skipping: Conditional result was False 2026-03-13 00:52:00.420993 | 2026-03-13 00:52:00.421098 | TASK [Update CA certs] 2026-03-13 00:52:00.955454 | hotstack | skipping: Conditional result was False 2026-03-13 00:52:00.963124 | 2026-03-13 00:52:00.963200 | TASK [Set _auth_type fact] 2026-03-13 00:52:01.014395 | hotstack | Output suppressed because no_log was given 2026-03-13 00:52:01.019282 | 2026-03-13 00:52:01.019349 | TASK [Write clouds.yaml - user/password] 2026-03-13 00:52:02.052711 | hotstack | Output suppressed because no_log was given 2026-03-13 00:52:02.064191 | 2026-03-13 00:52:02.064490 | TASK [Create application credential for sushy-emulator] 2026-03-13 00:52:04.117867 | hotstack | Output suppressed because no_log was given 2026-03-13 00:52:04.130195 | 2026-03-13 00:52:04.130337 | TASK [Read application credential from file] 2026-03-13 00:52:04.484921 | hotstack | Output suppressed because no_log was given 2026-03-13 00:52:04.496266 | 2026-03-13 00:52:04.496401 | TASK [Delete temporary application credential file] 2026-03-13 00:52:04.712458 | hotstack | Output suppressed because no_log was given 2026-03-13 00:52:04.717506 | 2026-03-13 00:52:04.717569 | TASK [Assert application credential id and secret] 2026-03-13 00:52:04.804210 | hotstack | Output suppressed because no_log was given 2026-03-13 00:52:04.815288 | 2026-03-13 00:52:04.815426 | TASK [Write new clouds.yaml with application credentials] 2026-03-13 00:52:05.544137 | hotstack | Output suppressed because no_log was given 2026-03-13 00:52:05.549232 | 2026-03-13 00:52:05.549300 | TASK [Write hotstack cloud-secret.yaml] 2026-03-13 00:52:06.198036 | hotstack | Output suppressed because no_log was given 2026-03-13 00:52:06.207436 | 2026-03-13 00:52:06.207575 | TASK [Write clouds.yaml with application credentials] 2026-03-13 00:52:06.240819 | 2026-03-13 00:52:06.240995 | TASK [Write hotstack cloud-secret.yaml] 2026-03-13 00:52:06.299768 | 2026-03-13 00:52:06.299877 | PLAY RECAP 2026-03-13 00:52:06.299919 | hotstack | ok: 18 changed: 11 unreachable: 0 failed: 0 skipped: 4 rescued: 0 ignored: 0 2026-03-13 00:52:06.299941 | 2026-03-13 00:52:06.393961 | PRE-RUN END RESULT_NORMAL: [trusted : review.rdoproject.org/config/playbooks/hotstack/pre_stage1.yaml@master] 2026-03-13 00:52:06.396130 | PRE-RUN START: [trusted : review.rdoproject.org/config/playbooks/hotstack/pre_stage2.yaml@master] 2026-03-13 00:52:06.964817 | 2026-03-13 00:52:06.964948 | PLAY [Create hotstack scenario] 2026-03-13 00:52:06.988513 | 2026-03-13 00:52:06.988641 | TASK [Assert scenario is defined] 2026-03-13 00:52:07.069301 | hotstack | ok: All assertions passed 2026-03-13 00:52:07.075931 | 2026-03-13 00:52:07.076004 | TASK [Fetch bootstrap vars from remote to zuul executor] 2026-03-13 00:52:07.640872 | hotstack | Output suppressed because no_log was given 2026-03-13 00:52:07.649850 | 2026-03-13 00:52:07.649960 | TASK [Load bootstrap vars with template resolution] 2026-03-13 00:52:07.684891 | hotstack | Output suppressed because no_log was given 2026-03-13 00:52:07.692161 | 2026-03-13 00:52:07.692303 | TASK [Fetch automation vars from remote to zuul executor] 2026-03-13 00:52:08.042733 | hotstack | Output suppressed because no_log was given 2026-03-13 00:52:08.049074 | 2026-03-13 00:52:08.049155 | TASK [Load automation vars with template resolution] 2026-03-13 00:52:08.139944 | hotstack | Output suppressed because no_log was given 2026-03-13 00:52:08.150576 | 2026-03-13 00:52:08.150750 | TASK [Slurp zuul-build-sshkey from file] 2026-03-13 00:52:08.552491 | hotstack | Output suppressed because no_log was given 2026-03-13 00:52:08.561302 | 2026-03-13 00:52:08.561410 | TASK [Set up infrastructure (Heat stack)] 2026-03-13 00:52:08.589092 | hotstack | Output suppressed because no_log was given 2026-03-13 00:52:08.647878 | 2026-03-13 00:52:08.648021 | TASK [dataplane_ssh_keys : Ensure Dataplane ssh keypair exists] 2026-03-13 00:52:09.330976 | hotstack | changed 2026-03-13 00:52:09.346392 | 2026-03-13 00:52:09.346564 | TASK [dataplane_ssh_keys : Ensure Nova migration ssh keypair exists] 2026-03-13 00:52:09.726271 | hotstack | changed 2026-03-13 00:52:09.739885 | 2026-03-13 00:52:09.740064 | TASK [dataplane_ssh_keys : Set SSH public key facts] 2026-03-13 00:52:09.780243 | hotstack | ok 2026-03-13 00:52:09.792984 | 2026-03-13 00:52:09.793159 | TASK [dataplane_ssh_keys : Store SSH public key facts in file] 2026-03-13 00:52:10.631617 | hotstack | changed 2026-03-13 00:52:10.655944 | 2026-03-13 00:52:10.656321 | TASK [heat_stack : Assert config is defined] 2026-03-13 00:52:10.772498 | hotstack | ok: All assertions passed 2026-03-13 00:52:10.779612 | 2026-03-13 00:52:10.779725 | TASK [heat_stack : Get latest snapset and update stack parameters] 2026-03-13 00:52:10.815508 | hotstack | skipping: Conditional result was False 2026-03-13 00:52:10.830355 | 2026-03-13 00:52:10.830503 | TASK [heat_stack : Debug latest snapset information] 2026-03-13 00:52:10.867850 | hotstack | skipping: Conditional result was False 2026-03-13 00:52:10.876509 | 2026-03-13 00:52:10.876597 | LOOP [heat_stack : Create tar archives and compress] 2026-03-13 00:52:10.912632 | 2026-03-13 00:52:10.912891 | TASK [heat_stack : Create stack] 2026-03-13 00:54:49.228499 | hotstack | changed 2026-03-13 00:54:49.245022 | 2026-03-13 00:54:49.245151 | TASK [heat_stack : Wait for stack to complete] 2026-03-13 00:54:50.904555 | hotstack | ok 2026-03-13 00:54:50.911786 | 2026-03-13 00:54:50.911906 | TASK [heat_stack : Stop if stack not complete] 2026-03-13 00:54:50.937058 | hotstack | skipping: Conditional result was False 2026-03-13 00:54:50.943377 | 2026-03-13 00:54:50.943449 | TASK [heat_stack : Get stack outputs] 2026-03-13 00:54:52.685522 | hotstack | changed 2026-03-13 00:54:52.692495 | 2026-03-13 00:54:52.692561 | TASK [heat_stack : Set stack outputs facts] 2026-03-13 00:54:52.735581 | hotstack | ok 2026-03-13 00:54:52.742965 | 2026-03-13 00:54:52.743039 | TASK [heat_stack : Store stack outputs in file] 2026-03-13 00:54:53.466093 | hotstack | changed 2026-03-13 00:54:53.481417 | 2026-03-13 00:54:53.481587 | TASK [Fetch stack outputs from remote to zuul executor] 2026-03-13 00:54:53.788002 | hotstack | Output suppressed because no_log was given 2026-03-13 00:54:53.801490 | 2026-03-13 00:54:53.801655 | TASK [Load stack output vars with template resolution] 2026-03-13 00:54:53.850807 | hotstack | Output suppressed because no_log was given 2026-03-13 00:54:53.862511 | 2026-03-13 00:54:53.862715 | TASK [Fetch cloud secrets from remote to zuul executor] 2026-03-13 00:54:54.210109 | hotstack | Output suppressed because no_log was given 2026-03-13 00:54:54.220789 | 2026-03-13 00:54:54.220923 | TASK [Load hotstack cloud secret vars with template resolution] 2026-03-13 00:54:54.258582 | hotstack | Output suppressed because no_log was given 2026-03-13 00:54:54.272135 | 2026-03-13 00:54:54.272292 | TASK [Add Hotstack controller-0 to ansible inventory] 2026-03-13 00:54:54.312368 | hotstack | Output suppressed because no_log was given 2026-03-13 00:54:54.323836 | 2026-03-13 00:54:54.324000 | TASK [Wait for Hotstack controller-0 to be ready] 2026-03-13 00:55:00.655637 | hotstack -> 38.102.83.243 | Output suppressed because no_log was given 2026-03-13 00:55:00.663843 | 2026-03-13 00:55:00.663950 | TASK [Start Zuul console on Hotstack controller-0] 2026-03-13 00:55:00.685999 | hotstack | Output suppressed because no_log was given 2026-03-13 00:55:00.721159 | 2026-03-13 00:55:00.721301 | TASK [start-zuul-console : Start zuul_console daemon.] 2026-03-13 00:55:01.197883 | hotstack -> 38.102.83.243 | ok 2026-03-13 00:55:01.222605 | 2026-03-13 00:55:01.222868 | TASK [Prepare the controller node] 2026-03-13 00:55:01.246745 | hotstack | Output suppressed because no_log was given 2026-03-13 00:55:01.331254 | 2026-03-13 00:55:01.331377 | TASK [controller : Assert config is defined] 2026-03-13 00:55:01.381764 | hotstack | ok: All assertions passed 2026-03-13 00:55:01.386891 | 2026-03-13 00:55:01.386960 | TASK [controller : Add controller-0 to the Ansible inventory] 2026-03-13 00:55:01.416495 | hotstack | ok 2026-03-13 00:55:01.422452 | 2026-03-13 00:55:01.422557 | TASK [controller : Read Dataplane SSH private key content] 2026-03-13 00:55:01.673974 | hotstack | ok 2026-03-13 00:55:01.687159 | 2026-03-13 00:55:01.687302 | TASK [controller : Read Nova migration SSH private key content] 2026-03-13 00:55:01.929325 | hotstack | ok 2026-03-13 00:55:01.936499 | 2026-03-13 00:55:01.936576 | TASK [controller : Wait for controller-0 to be ready] 2026-03-13 00:55:02.747622 | hotstack -> 38.102.83.243 | ok 2026-03-13 00:55:02.761785 | 2026-03-13 00:55:02.761929 | LOOP [controller : Ensure directories exists] 2026-03-13 00:55:03.371373 | hotstack -> 38.102.83.243 | changed: "/home/zuul/data" 2026-03-13 00:55:03.371859 | 2026-03-13 00:55:03.829780 | hotstack -> 38.102.83.243 | changed: "/home/zuul/bin" 2026-03-13 00:55:03.842806 | 2026-03-13 00:55:03.842965 | TASK [controller : Setup NFS server] 2026-03-13 00:55:03.918824 | hotstack | ok 2026-03-13 00:55:03.931979 | hotstack | included: /var/lib/zuul/builds/46a5d1853f074c73b1d21511708ab398/trusted/project_3/github.com/openstack-k8s-operators/hotstack/roles/controller/tasks/setup_nfs.yml 2026-03-13 00:55:03.946063 | 2026-03-13 00:55:03.946216 | LOOP [controller : Create NFS export directories] 2026-03-13 00:55:04.533126 | hotstack -> 38.102.83.243 | changed: 2026-03-13 00:55:04.533404 | hotstack -> 38.102.83.243 | { 2026-03-13 00:55:04.533442 | hotstack -> 38.102.83.243 | "mode": "0777", 2026-03-13 00:55:04.533465 | hotstack -> 38.102.83.243 | "path": "/export/nova-console-recordings" 2026-03-13 00:55:04.533487 | hotstack -> 38.102.83.243 | } 2026-03-13 00:55:04.547267 | 2026-03-13 00:55:04.547385 | TASK [controller : Configure NFS exports] 2026-03-13 00:55:05.635972 | hotstack -> 38.102.83.243 | changed 2026-03-13 00:55:05.644500 | 2026-03-13 00:55:05.644616 | TASK [controller : Enable and start NFS server] 2026-03-13 00:55:07.865517 | hotstack -> 38.102.83.243 | changed 2026-03-13 00:55:07.884277 | 2026-03-13 00:55:07.884417 | TASK [controller : Reload NFS exports] 2026-03-13 00:55:08.478117 | hotstack -> 38.102.83.243 | ok 2026-03-13 00:55:08.491452 | 2026-03-13 00:55:08.491612 | LOOP [controller : Add common functions to the bin_dir] 2026-03-13 00:55:09.366242 | hotstack -> 38.102.83.243 | changed: "bin/hotstack-common-funcs.sh" 2026-03-13 00:55:09.366808 | hotstack -> 38.102.83.243 | changed: All items complete 2026-03-13 00:55:09.366910 | 2026-03-13 00:55:09.382964 | 2026-03-13 00:55:09.383166 | LOOP [controller : Add bin utilities to the bin_dir] 2026-03-13 00:55:10.674516 | hotstack -> 38.102.83.243 | changed: "bin/hotstack-approve-install-plan" 2026-03-13 00:55:11.673838 | hotstack -> 38.102.83.243 | changed: "bin/hotstack-collect-tempest-logs" 2026-03-13 00:55:12.516700 | hotstack -> 38.102.83.243 | changed: "bin/hotstack-leader-election-tune" 2026-03-13 00:55:13.124502 | hotstack -> 38.102.83.243 | changed: "bin/hotstack-nova-discover-hosts" 2026-03-13 00:55:13.759848 | hotstack -> 38.102.83.243 | changed: "bin/hotstack-snapset" 2026-03-13 00:55:14.336594 | hotstack -> 38.102.83.243 | changed: "bin/hotstack-openstack-version-patch" 2026-03-13 00:55:14.928730 | hotstack -> 38.102.83.243 | changed: "bin/hotstack-wait-for-bmh" 2026-03-13 00:55:14.944743 | 2026-03-13 00:55:14.944866 | TASK [controller : Write ansible inventory to file on controller-0] 2026-03-13 00:55:15.003156 | hotstack | skipping: Conditional result was False 2026-03-13 00:55:15.011249 | 2026-03-13 00:55:15.011349 | TASK [controller : Copy Dataplane ssh private key to controller] 2026-03-13 00:55:15.606759 | hotstack -> 38.102.83.243 | changed 2026-03-13 00:55:15.622268 | 2026-03-13 00:55:15.622414 | TASK [controller : Copy Dataplane ssh public key to controller] 2026-03-13 00:55:16.291331 | hotstack -> 38.102.83.243 | changed 2026-03-13 00:55:16.305707 | 2026-03-13 00:55:16.305851 | TASK [controller : Copy Nova migrate ssh private key to controller] 2026-03-13 00:55:16.981309 | hotstack -> 38.102.83.243 | changed 2026-03-13 00:55:16.998996 | 2026-03-13 00:55:16.999155 | TASK [controller : Copy Nova migrate ssh public key to controller] 2026-03-13 00:55:17.743177 | hotstack -> 38.102.83.243 | changed 2026-03-13 00:55:17.751776 | 2026-03-13 00:55:17.751876 | TASK [controller : Copy dataplane authorized keys to controller] 2026-03-13 00:55:18.442757 | hotstack -> 38.102.83.243 | changed 2026-03-13 00:55:18.457347 | 2026-03-13 00:55:18.457486 | TASK [controller : Ensure cloud config directory exists] 2026-03-13 00:55:18.803155 | hotstack -> 38.102.83.243 | changed 2026-03-13 00:55:18.810498 | 2026-03-13 00:55:18.810563 | TASK [controller : Template the clouds.yaml] 2026-03-13 00:55:19.563660 | hotstack -> 38.102.83.243 | Output suppressed because no_log was given 2026-03-13 00:55:19.577841 | 2026-03-13 00:55:19.578002 | TASK [controller : Write CA certificate to file] 2026-03-13 00:55:20.332922 | hotstack -> 38.102.83.243 | Output suppressed because no_log was given 2026-03-13 00:55:20.347020 | 2026-03-13 00:55:20.347214 | TASK [controller : Write Ironic nodes YAML] 2026-03-13 00:55:20.396487 | hotstack | skipping: Conditional result was False 2026-03-13 00:55:20.410197 | 2026-03-13 00:55:20.410377 | TASK [controller : Install OpenStack client] 2026-03-13 00:55:20.459034 | hotstack | skipping: Conditional result was False 2026-03-13 00:55:20.477265 | 2026-03-13 00:55:20.477417 | TASK [Install Openshift] 2026-03-13 00:55:20.500153 | hotstack | Output suppressed because no_log was given 2026-03-13 00:55:20.553601 | 2026-03-13 00:55:20.553740 | TASK [ocp_agent_installer : Skip OCP agent installer for MicroShift] 2026-03-13 00:55:20.587871 | hotstack | skipping: Conditional result was False 2026-03-13 00:55:20.593497 | 2026-03-13 00:55:20.593689 | TASK [ocp_agent_installer : Assert config is defined] 2026-03-13 00:55:20.712808 | hotstack -> 38.102.83.243 | ok: All assertions passed 2026-03-13 00:55:20.772590 | 2026-03-13 00:55:20.772778 | TASK [ocp_agent_installer : Install using agent installer] 2026-03-13 00:55:20.845722 | hotstack | ok 2026-03-13 00:55:20.864880 | hotstack | included: /var/lib/zuul/builds/46a5d1853f074c73b1d21511708ab398/trusted/project_3/github.com/openstack-k8s-operators/hotstack/roles/ocp_agent_installer/tasks/_install.yml 2026-03-13 00:55:20.875376 | 2026-03-13 00:55:20.875474 | TASK [ocp_agent_installer : Assert config is defined] 2026-03-13 00:55:20.961287 | hotstack -> 38.102.83.243 | ok: All assertions passed 2026-03-13 00:55:20.968689 | 2026-03-13 00:55:20.968770 | LOOP [ocp_agent_installer : Ensure directory exists] 2026-03-13 00:55:21.248566 | hotstack -> 38.102.83.243 | ok: "/home/zuul/bin" 2026-03-13 00:55:21.248947 | hotstack -> 38.102.83.243 | changed: All items complete 2026-03-13 00:55:21.248997 | 2026-03-13 00:55:21.506796 | hotstack -> 38.102.83.243 | changed: "/home/zuul/.kube" 2026-03-13 00:55:21.792017 | hotstack -> 38.102.83.243 | changed: "/home/zuul/ocp-cluster" 2026-03-13 00:55:22.078751 | hotstack -> 38.102.83.243 | changed: "/home/zuul/ocp-cluster/openshift" 2026-03-13 00:55:22.410903 | hotstack -> 38.102.83.243 | changed: "/home/zuul/agent-installer" 2026-03-13 00:55:22.677059 | hotstack -> 38.102.83.243 | changed: "/home/zuul/cluster-custom-config/" 2026-03-13 00:55:22.957928 | hotstack -> 38.102.83.243 | changed: "/home/zuul/cluster-custom-config//butane" 2026-03-13 00:55:23.222409 | hotstack -> 38.102.83.243 | changed: "/home/zuul/cluster-custom-config//machine-configs" 2026-03-13 00:55:23.478536 | hotstack -> 38.102.83.243 | changed: "/home/zuul/cluster-custom-config//config-assets" 2026-03-13 00:55:23.506377 | 2026-03-13 00:55:23.506563 | TASK [ocp_agent_installer : Install package requirements for agent installer] 2026-03-13 00:55:37.064956 | hotstack -> 38.102.83.243 | ok: Nothing to do 2026-03-13 00:55:37.082891 | 2026-03-13 00:55:37.083109 | TASK [ocp_agent_installer : Run tasks/install_client.yml] 2026-03-13 00:55:37.127219 | hotstack | ok 2026-03-13 00:55:37.147329 | hotstack | included: /var/lib/zuul/builds/46a5d1853f074c73b1d21511708ab398/trusted/project_3/github.com/openstack-k8s-operators/hotstack/roles/ocp_agent_installer/tasks/install_client.yml 2026-03-13 00:55:37.166697 | 2026-03-13 00:55:37.166876 | TASK [ocp_agent_installer : Download the client] 2026-03-13 00:55:40.769150 | hotstack -> 38.102.83.243 | changed: OK (73357262 bytes) 2026-03-13 00:55:40.779614 | 2026-03-13 00:55:40.779755 | TASK [ocp_agent_installer : Extract client to /home/zuul/bin] 2026-03-13 00:55:45.060629 | hotstack -> 38.102.83.243 | changed 2026-03-13 00:55:45.075169 | 2026-03-13 00:55:45.075332 | TASK [ocp_agent_installer : Configure bash completion] 2026-03-13 00:55:45.613580 | hotstack -> 38.102.83.243 | changed 2026-03-13 00:55:45.629204 | 2026-03-13 00:55:45.629368 | TASK [ocp_agent_installer : Run tasks/install_installer.yml] 2026-03-13 00:55:45.672986 | hotstack | ok 2026-03-13 00:55:45.688794 | hotstack | included: /var/lib/zuul/builds/46a5d1853f074c73b1d21511708ab398/trusted/project_3/github.com/openstack-k8s-operators/hotstack/roles/ocp_agent_installer/tasks/install_installer.yml 2026-03-13 00:55:45.709694 | 2026-03-13 00:55:45.709798 | TASK [ocp_agent_installer : Download the installer] 2026-03-13 00:55:56.330429 | hotstack -> 38.102.83.243 | changed: OK (437747386 bytes) 2026-03-13 00:55:56.346549 | 2026-03-13 00:55:56.346780 | TASK [ocp_agent_installer : Extract installer to /home/zuul/bin] 2026-03-13 00:56:11.426809 | hotstack -> 38.102.83.243 | changed 2026-03-13 00:56:11.441724 | 2026-03-13 00:56:11.441890 | TASK [ocp_agent_installer : Write OCP install-config.yaml] 2026-03-13 00:56:12.194322 | hotstack -> 38.102.83.243 | changed 2026-03-13 00:56:12.207646 | 2026-03-13 00:56:12.207822 | TASK [ocp_agent_installer : Replace pull secret in install-config.yaml] 2026-03-13 00:56:12.725746 | hotstack -> 38.102.83.243 | Output suppressed because no_log was given 2026-03-13 00:56:12.741773 | 2026-03-13 00:56:12.741961 | TASK [ocp_agent_installer : Write OCP agent-config.yaml] 2026-03-13 00:56:13.461346 | hotstack -> 38.102.83.243 | changed 2026-03-13 00:56:13.477233 | 2026-03-13 00:56:13.477406 | TASK [ocp_agent_installer : Include tasks to generate machine configs] 2026-03-13 00:56:13.522842 | hotstack | ok 2026-03-13 00:56:13.537407 | hotstack | included: /var/lib/zuul/builds/46a5d1853f074c73b1d21511708ab398/trusted/project_3/github.com/openstack-k8s-operators/hotstack/roles/ocp_agent_installer/tasks/machine_configs.yml 2026-03-13 00:56:13.558426 | 2026-03-13 00:56:13.558525 | LOOP [ocp_agent_installer : Template butane config for net.ifnames] 2026-03-13 00:56:14.385365 | hotstack -> 38.102.83.243 | changed: "master" 2026-03-13 00:56:14.408067 | 2026-03-13 00:56:14.408220 | LOOP [ocp_agent_installer : Generate MachineConfig for iscsi] 2026-03-13 00:56:14.797149 | hotstack -> 38.102.83.243 | changed 2026-03-13 00:56:14.797660 | hotstack -> 38.102.83.243 | changed: All items complete 2026-03-13 00:56:14.797803 | 2026-03-13 00:56:14.818992 | 2026-03-13 00:56:14.819342 | LOOP [ocp_agent_installer : Template butane config for iscsi] 2026-03-13 00:56:15.484999 | hotstack -> 38.102.83.243 | changed: "master" 2026-03-13 00:56:15.505341 | 2026-03-13 00:56:15.505583 | LOOP [ocp_agent_installer : Generate MachineConfig for iscsi] 2026-03-13 00:56:15.903396 | hotstack -> 38.102.83.243 | changed 2026-03-13 00:56:15.903883 | hotstack -> 38.102.83.243 | changed: All items complete 2026-03-13 00:56:15.903948 | 2026-03-13 00:56:15.918425 | 2026-03-13 00:56:15.918522 | LOOP [ocp_agent_installer : Template butane config for multipath] 2026-03-13 00:56:16.554277 | hotstack -> 38.102.83.243 | changed: "master" 2026-03-13 00:56:16.572001 | 2026-03-13 00:56:16.572196 | LOOP [ocp_agent_installer : Generate MachineConfig for multipath] 2026-03-13 00:56:17.029705 | hotstack -> 38.102.83.243 | changed 2026-03-13 00:56:17.030146 | hotstack -> 38.102.83.243 | changed: All items complete 2026-03-13 00:56:17.030361 | 2026-03-13 00:56:17.053952 | 2026-03-13 00:56:17.054158 | LOOP [ocp_agent_installer : Template butane config for LVM cinder-volumes] 2026-03-13 00:56:17.845973 | hotstack -> 38.102.83.243 | changed: "master" 2026-03-13 00:56:17.865060 | 2026-03-13 00:56:17.865298 | LOOP [ocp_agent_installer : Generate MachineConfig for multipath] 2026-03-13 00:56:18.216448 | hotstack -> 38.102.83.243 | changed 2026-03-13 00:56:18.216926 | hotstack -> 38.102.83.243 | changed: All items complete 2026-03-13 00:56:18.216991 | 2026-03-13 00:56:18.235883 | 2026-03-13 00:56:18.236094 | TASK [ocp_agent_installer : Include tasks to generate config assets] 2026-03-13 00:56:18.286874 | hotstack | ok 2026-03-13 00:56:18.306616 | hotstack | included: /var/lib/zuul/builds/46a5d1853f074c73b1d21511708ab398/trusted/project_3/github.com/openstack-k8s-operators/hotstack/roles/ocp_agent_installer/tasks/config_assets.yml 2026-03-13 00:56:18.336984 | 2026-03-13 00:56:18.337115 | TASK [ocp_agent_installer : Template ovn-k8s customization] 2026-03-13 00:56:19.055864 | hotstack -> 38.102.83.243 | changed 2026-03-13 00:56:19.070492 | 2026-03-13 00:56:19.070711 | TASK [ocp_agent_installer : Copy Etcd customization] 2026-03-13 00:56:19.793940 | hotstack -> 38.102.83.243 | changed 2026-03-13 00:56:19.809871 | 2026-03-13 00:56:19.810027 | TASK [ocp_agent_installer : Template ImageContentSourcePolicy customization] 2026-03-13 00:56:19.880317 | hotstack | skipping: Conditional result was False 2026-03-13 00:56:19.891339 | 2026-03-13 00:56:19.891529 | TASK [ocp_agent_installer : Initialize _ocp_additional_trusted_ca_map fact] 2026-03-13 00:56:19.947208 | hotstack | skipping: Conditional result was False 2026-03-13 00:56:19.954280 | 2026-03-13 00:56:19.954348 | LOOP [ocp_agent_installer : Append to _ocp_additional_trusted_ca_map fact] 2026-03-13 00:56:20.002839 | hotstack | skipping: Conditional result was False 2026-03-13 00:56:20.030858 | 2026-03-13 00:56:20.031091 | TASK [ocp_agent_installer : Template additional CA config map] 2026-03-13 00:56:20.081835 | hotstack | skipping: Conditional result was False 2026-03-13 00:56:20.102535 | 2026-03-13 00:56:20.102786 | TASK [ocp_agent_installer : Template Image config (additionalTrustedCA and/or insecureRegistries)] 2026-03-13 00:56:20.162867 | hotstack | skipping: Conditional result was False 2026-03-13 00:56:20.176233 | 2026-03-13 00:56:20.176380 | TASK [ocp_agent_installer : Copy machine_configs to manifests dir] 2026-03-13 00:56:20.583038 | hotstack -> 38.102.83.243 | changed 2026-03-13 00:56:20.598117 | 2026-03-13 00:56:20.598307 | TASK [ocp_agent_installer : Copy config assets to manifests dir] 2026-03-13 00:56:21.025807 | hotstack -> 38.102.83.243 | changed 2026-03-13 00:56:21.043622 | 2026-03-13 00:56:21.043821 | TASK [ocp_agent_installer : Run tasks/pxe_assets.yml] 2026-03-13 00:56:21.082553 | hotstack | ok 2026-03-13 00:56:21.095497 | hotstack | included: /var/lib/zuul/builds/46a5d1853f074c73b1d21511708ab398/trusted/project_3/github.com/openstack-k8s-operators/hotstack/roles/ocp_agent_installer/tasks/pxe_assets.yml 2026-03-13 00:56:21.128116 | 2026-03-13 00:56:21.128242 | TASK [ocp_agent_installer : Create the boot-artifacts directory] 2026-03-13 00:56:21.470837 | hotstack -> 38.102.83.243 | changed 2026-03-13 00:56:21.485648 | 2026-03-13 00:56:21.485854 | TASK [ocp_agent_installer : Create PXE assets] 2026-03-12 20:56:23.257404 | controller-0 | level=warning msg=Compute[0].Hyperthreading: Disabled is ignored 2026-03-12 20:56:23.943737 | controller-0 | level=warning msg=ControlPlane.Hyperthreading: Disabled is ignored 2026-03-12 20:56:23.943778 | controller-0 | level=info msg=Configuration has 1 master replicas and 0 worker replicas 2026-03-12 20:56:23.943792 | controller-0 | level=info msg=The rendezvous host IP (node0 IP) is 192.168.32.10 2026-03-12 20:56:24.599452 | controller-0 | level=info msg=Extracting base ISO from release payload 2026-03-12 20:57:02.032011 | controller-0 | level=info msg=Base ISO obtained from release and cached at [/home/zuul/.cache/agent/image_cache/coreos-x86_64.iso] 2026-03-12 20:58:03.473785 | controller-0 | level=info msg=Consuming Extra Manifests from target directory 2026-03-12 20:58:03.474166 | controller-0 | level=info msg=Consuming Install Config from target directory 2026-03-12 20:58:04.008827 | controller-0 | level=info msg=Consuming Agent Config from target directory 2026-03-12 20:58:04.008891 | controller-0 | level=info msg=Created iPXE script agent.x86_64.ipxe in boot-artifacts directory 2026-03-12 20:58:04.009004 | controller-0 | level=info msg=PXE boot artifacts created in: boot-artifacts 2026-03-12 20:58:04.009090 | controller-0 | level=info msg=Kernel parameters for PXE boot: rw ignition.firstboot ignition.platform.id=metal 2026-03-13 00:58:04.462585 | hotstack -> 38.102.83.243 | changed 2026-03-13 00:58:04.476135 | 2026-03-13 00:58:04.476293 | TASK [ocp_agent_installer : Set serial console in ipxe] 2026-03-13 00:58:05.128041 | hotstack -> 38.102.83.243 | changed: line replaced 2026-03-13 00:58:05.142378 | 2026-03-13 00:58:05.142570 | TASK [ocp_agent_installer : Disable net.ifnames] 2026-03-13 00:58:05.671744 | hotstack -> 38.102.83.243 | changed: line replaced 2026-03-13 00:58:05.685778 | 2026-03-13 00:58:05.685926 | LOOP [ocp_agent_installer : Copy boot-artifacts to the web server - (ocp_agent_installer_boot_artifacts_dir)] 2026-03-13 00:58:06.498443 | hotstack -> 38.102.83.243 | changed: "agent.x86_64-vmlinuz" 2026-03-13 00:58:06.499083 | 2026-03-13 00:58:07.680102 | hotstack -> 38.102.83.243 | changed: "agent.x86_64-initrd.img" 2026-03-13 00:58:34.398401 | hotstack -> 38.102.83.243 | changed: "agent.x86_64-rootfs.img" 2026-03-13 00:58:34.810179 | hotstack -> 38.102.83.243 | changed: "agent.x86_64.ipxe" 2026-03-13 00:58:34.829964 | 2026-03-13 00:58:34.830145 | TASK [ocp_agent_installer : Run tasks/iso_assets.yml] 2026-03-13 00:58:34.867612 | hotstack | skipping: Conditional result was False 2026-03-13 00:58:34.881199 | 2026-03-13 00:58:34.881376 | TASK [ocp_agent_installer : Copy auth/kubeconfig to ~/.kube/config] 2026-03-13 00:58:35.339233 | hotstack -> 38.102.83.243 | changed 2026-03-13 00:58:35.348027 | 2026-03-13 00:58:35.348152 | TASK [ocp_agent_installer : Wait for bootstrap-complete] 2026-03-12 20:59:07.815955 | controller-0 | level=info msg=Waiting for cluster install to initialize. Sleeping for 30 seconds 2026-03-12 20:59:39.819486 | controller-0 | level=info msg=Waiting for cluster install to initialize. Sleeping for 30 seconds 2026-03-12 21:00:11.823138 | controller-0 | level=info msg=Waiting for cluster install to initialize. Sleeping for 30 seconds 2026-03-12 21:00:43.829319 | controller-0 | level=info msg=Cannot access Rendezvous Host. There may be a network configuration problem, check console for additional info 2026-03-12 21:01:15.832378 | controller-0 | level=info msg=Waiting for cluster install to initialize. Sleeping for 30 seconds 2026-03-12 21:01:47.857562 | controller-0 | level=info msg=Cluster is not ready for install. Check validations 2026-03-12 21:01:47.862421 | controller-0 | level=info msg=Host 97fa34ad-777e-4bac-b915-c705721d411c: Successfully registered 2026-03-12 21:01:51.887164 | controller-0 | level=warning msg=Cluster validation: The cluster has hosts that are not ready to install. 2026-03-12 21:01:57.388679 | controller-0 | level=warning msg=Host master-0 validation: Host couldn't synchronize with any NTP server 2026-03-12 21:01:57.396293 | controller-0 | level=warning msg=Host master-0: updated status from discovering to insufficient (Host cannot be installed due to following failing validation(s): Host couldn't synchronize with any NTP server) 2026-03-12 21:02:51.837903 | controller-0 | level=info msg=Host master-0 validation: Host NTP is synced 2026-03-12 21:02:51.841098 | controller-0 | level=info msg=Host master-0: updated status from insufficient to known (Host is ready to be installed) 2026-03-12 21:03:01.917864 | controller-0 | level=info msg=Cluster is ready for install 2026-03-12 21:03:01.918292 | controller-0 | level=info msg=Cluster validation: All hosts in the cluster are ready to install. 2026-03-12 21:03:03.933342 | controller-0 | level=info msg=Preparing cluster for installation 2026-03-12 21:03:07.979027 | controller-0 | level=info msg=Host master-0: updated status from known to preparing-for-installation (Host finished successfully to prepare for installation) 2026-03-12 21:04:06.762107 | controller-0 | level=info msg=Host master-0: New image status quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4c60342a9b9efba628155b01cd76cba0b9d54616013fcdf044fafe7700b09d2c. result: success. time: 1.80 seconds; size: 453.71 Megabytes; download rate: 263.95 MBps 2026-03-12 21:04:10.793963 | controller-0 | level=info msg=Host master-0: updated status from preparing-for-installation to preparing-successful (Host finished successfully to prepare for installation) 2026-03-12 21:04:21.002720 | controller-0 | level=info msg=Cluster installation in progress 2026-03-12 21:04:27.053039 | controller-0 | level=info msg=Host master-0: updated status from preparing-successful to installing (Installation is in progress) 2026-03-12 21:04:51.256835 | controller-0 | level=info msg=Host: master-0, reached installation stage Installing: bootstrap 2026-03-12 21:04:57.312019 | controller-0 | level=info msg=Host: master-0, reached installation stage Waiting for bootkube 2026-03-12 21:06:14.325310 | controller-0 | level=info msg=Bootstrap Kube API Initialized 2026-03-12 21:08:58.044929 | controller-0 | level=info msg=Host: master-0, reached installation stage Writing image to disk 2026-03-12 21:09:00.059023 | controller-0 | level=info msg=Host: master-0, reached installation stage Writing image to disk: 14% 2026-03-12 21:09:04.093325 | controller-0 | level=info msg=Host: master-0, reached installation stage Writing image to disk: 21% 2026-03-12 21:09:06.109527 | controller-0 | level=info msg=Host: master-0, reached installation stage Writing image to disk: 28% 2026-03-12 21:09:08.123976 | controller-0 | level=info msg=Host: master-0, reached installation stage Writing image to disk: 33% 2026-03-12 21:09:10.141023 | controller-0 | level=info msg=Host: master-0, reached installation stage Writing image to disk: 38% 2026-03-12 21:09:12.155738 | controller-0 | level=info msg=Host: master-0, reached installation stage Writing image to disk: 51% 2026-03-12 21:09:16.187860 | controller-0 | level=info msg=Host: master-0, reached installation stage Writing image to disk: 56% 2026-03-12 21:09:18.206804 | controller-0 | level=info msg=Host: master-0, reached installation stage Writing image to disk: 62% 2026-03-12 21:09:20.225153 | controller-0 | level=info msg=Host: master-0, reached installation stage Writing image to disk: 67% 2026-03-12 21:09:22.248804 | controller-0 | level=info msg=Host: master-0, reached installation stage Writing image to disk: 72% 2026-03-12 21:09:24.267117 | controller-0 | level=info msg=Host: master-0, reached installation stage Writing image to disk: 79% 2026-03-12 21:09:26.283466 | controller-0 | level=info msg=Host: master-0, reached installation stage Writing image to disk: 84% 2026-03-12 21:09:28.301863 | controller-0 | level=info msg=Host: master-0, reached installation stage Writing image to disk: 89% 2026-03-12 21:09:30.320937 | controller-0 | level=info msg=Host: master-0, reached installation stage Writing image to disk: 100% 2026-03-12 21:11:32.220028 | controller-0 | level=info msg=Bootstrap configMap status is complete 2026-03-12 21:11:34.220591 | controller-0 | level=info msg=Bootstrap is complete 2026-03-12 21:11:34.220821 | controller-0 | level=info msg=cluster bootstrap is complete 2026-03-13 01:11:34.543041 | hotstack -> 38.102.83.243 | changed 2026-03-13 01:11:34.557521 | 2026-03-13 01:11:34.557713 | TASK [ocp_agent_installer : Wait for install-complete] 2026-03-12 21:11:35.247080 | controller-0 | level=info msg=Bootstrap Kube API Initialized 2026-03-12 21:11:35.250688 | controller-0 | level=info msg=Bootstrap configMap status is complete 2026-03-12 21:11:37.253012 | controller-0 | level=info msg=Bootstrap is complete 2026-03-12 21:11:37.253116 | controller-0 | level=info msg=cluster bootstrap is complete 2026-03-12 21:36:20.407453 | controller-0 | level=info msg=Cluster is installed 2026-03-12 21:36:20.409439 | controller-0 | level=info msg=Install complete! 2026-03-12 21:36:20.409584 | controller-0 | level=info msg=To access the cluster as the system:admin user when using 'oc', run 2026-03-12 21:36:20.409629 | controller-0 | level=info msg= export KUBECONFIG=/home/zuul/ocp-cluster/auth/kubeconfig 2026-03-12 21:36:20.409718 | controller-0 | level=info msg=Access the OpenShift web-console here: https://console-openshift-console.apps.sno.openstack.lab 2026-03-12 21:36:20.409817 | controller-0 | level=info msg=Login to the console with user: "kubeadmin", and password: "NA2sj-ANKHI-sFNHv-FrXQY" 2026-03-13 01:36:20.749027 | hotstack -> 38.102.83.243 | changed 2026-03-13 01:36:20.762036 | 2026-03-13 01:36:20.762173 | TASK [ocp_agent_installer : Add ingress certificate to CA trust] 2026-03-13 01:36:20.843073 | hotstack | ok 2026-03-13 01:36:20.858317 | hotstack | included: /var/lib/zuul/builds/46a5d1853f074c73b1d21511708ab398/trusted/project_3/github.com/openstack-k8s-operators/hotstack/roles/ocp_agent_installer/tasks/ingress_cert.yml 2026-03-13 01:36:20.900884 | 2026-03-13 01:36:20.900973 | TASK [ocp_agent_installer : Extract ingress CA cert] 2026-03-13 01:36:22.428485 | hotstack -> 38.102.83.243 | Output suppressed because no_log was given 2026-03-13 01:36:22.440938 | 2026-03-13 01:36:22.441073 | TASK [ocp_agent_installer : Write ingress cert to ca-trust] 2026-03-13 01:36:23.488177 | hotstack -> 38.102.83.243 | changed 2026-03-13 01:36:23.502220 | 2026-03-13 01:36:23.502355 | TASK [ocp_agent_installer : Update CA trust] 2026-03-13 01:36:26.545557 | hotstack -> 38.102.83.243 | changed 2026-03-13 01:36:26.558379 | 2026-03-13 01:36:26.558510 | TASK [ocp_agent_installer : Wait for 25 hours to prepare for snapshot] 2026-03-13 01:36:26.626461 | hotstack | skipping: Conditional result was False 2026-03-13 01:36:26.639095 | 2026-03-13 01:36:26.639313 | TASK [ocp_agent_installer : Prepare for Snapshot - and shutdown] 2026-03-13 01:36:27.185687 | hotstack | skipping: Conditional result was False 2026-03-13 01:36:27.198485 | 2026-03-13 01:36:27.198644 | TASK [ocp_agent_installer : Wait for cluster, uncordon, wait for cluster, wait for route API version ...] 2026-03-13 01:36:27.743311 | hotstack | skipping: Conditional result was False 2026-03-13 01:36:27.768162 | 2026-03-13 01:36:27.768306 | TASK [Set up RedFish Virtual bmc] 2026-03-13 01:36:27.849734 | 2026-03-13 01:36:27.850009 | LOOP [Remove the cloud-secrets.yaml file from zuul executor] 2026-03-13 01:36:28.144102 | hotstack | Output suppressed because no_log was given 2026-03-13 01:36:28.150769 | 2026-03-13 01:36:28.150851 | LOOP [Remove the cloud-secret files] 2026-03-13 01:36:28.886107 | hotstack | Output suppressed because no_log was given 2026-03-13 01:36:28.905795 | 2026-03-13 01:36:28.905889 | PLAY RECAP 2026-03-13 01:36:28.905933 | hotstack | ok: 88 changed: 59 unreachable: 0 failed: 0 skipped: 18 rescued: 0 ignored: 0 2026-03-13 01:36:28.905953 | 2026-03-13 01:36:29.033734 | PRE-RUN END RESULT_NORMAL: [trusted : review.rdoproject.org/config/playbooks/hotstack/pre_stage2.yaml@master] 2026-03-13 01:36:29.035607 | PRE-RUN START: [untrusted : github.com/openstack-k8s-operators/hotstack/ci/playbooks/pre-deploy.yml@main] 2026-03-13 01:36:29.583751 | 2026-03-13 01:36:29.583872 | PLAY [Configure content provider insecure registry on hotstack OCP cluster] 2026-03-13 01:36:29.606025 | 2026-03-13 01:36:29.606110 | TASK [Set content provider registry] 2026-03-13 01:36:29.640625 | hotstack | skipping: Conditional result was False 2026-03-13 01:36:29.646251 | 2026-03-13 01:36:29.646323 | TASK [Fetch stack outputs from remote to zuul executor] 2026-03-13 01:36:29.685224 | 2026-03-13 01:36:29.685379 | TASK [Load stack output vars] 2026-03-13 01:36:29.725097 | 2026-03-13 01:36:29.725217 | TASK [Add Hotstack controller-0 to ansible inventory] 2026-03-13 01:36:29.762036 | 2026-03-13 01:36:29.762280 | TASK [Wait for Hotstack controller-0 to be ready] 2026-03-13 01:36:29.814126 | 2026-03-13 01:36:29.814352 | TASK [Patch image.config to add insecure and allowed registries] 2026-03-13 01:36:29.843655 | hotstack | skipping: Conditional result was False 2026-03-13 01:36:29.855874 | 2026-03-13 01:36:29.856006 | TASK [Wait for MachineConfigPool to start updating] 2026-03-13 01:36:29.893600 | hotstack | skipping: Conditional result was False 2026-03-13 01:36:29.901017 | 2026-03-13 01:36:29.901108 | TASK [Wait for MachineConfigPool update to complete] 2026-03-13 01:36:29.936605 | hotstack | skipping: Conditional result was False 2026-03-13 01:36:29.948868 | 2026-03-13 01:36:29.949006 | TASK [Wait for API server to become available] 2026-03-13 01:36:29.978266 | hotstack | skipping: Conditional result was False 2026-03-13 01:36:29.989905 | 2026-03-13 01:36:29.990035 | TASK [Wait for all nodes to be Ready] 2026-03-13 01:36:30.019171 | hotstack | skipping: Conditional result was False 2026-03-13 01:36:30.031906 | 2026-03-13 01:36:30.032057 | TASK [Wait for cluster operators to stabilize] 2026-03-13 01:36:30.060574 | hotstack | skipping: Conditional result was False 2026-03-13 01:36:30.073591 | 2026-03-13 01:36:30.073784 | TASK [Verify insecure registry is configured] 2026-03-13 01:36:30.111540 | hotstack | skipping: Conditional result was False 2026-03-13 01:36:30.127448 | 2026-03-13 01:36:30.127577 | TASK [Log configured insecure registries] 2026-03-13 01:36:30.155165 | hotstack | skipping: Conditional result was False 2026-03-13 01:36:30.195178 | 2026-03-13 01:36:30.195265 | PLAY RECAP 2026-03-13 01:36:30.195324 | hotstack | ok: 0 changed: 0 unreachable: 0 failed: 0 skipped: 13 rescued: 0 ignored: 0 2026-03-13 01:36:30.195353 | 2026-03-13 01:36:30.301597 | PRE-RUN END RESULT_NORMAL: [untrusted : github.com/openstack-k8s-operators/hotstack/ci/playbooks/pre-deploy.yml@main] 2026-03-13 01:36:30.305598 | RUN START: [untrusted : github.com/openstack-k8s-operators/hotstack/ci/playbooks/run-deploy.yml@main] 2026-03-13 01:36:30.854494 | 2026-03-13 01:36:30.854617 | PLAY [Create hotstack scenario] 2026-03-13 01:36:30.875954 | 2026-03-13 01:36:30.876033 | TASK [Fetch bootstrap vars from remote to zuul executor] 2026-03-13 01:36:31.559787 | hotstack | ok 2026-03-13 01:36:31.571606 | 2026-03-13 01:36:31.571821 | TASK [Load bootstrap vars with template resolution] 2026-03-13 01:36:31.619877 | hotstack | ok 2026-03-13 01:36:31.632956 | 2026-03-13 01:36:31.633087 | TASK [Fetch automation vars from remote to zuul executor] 2026-03-13 01:36:32.037478 | hotstack | ok 2026-03-13 01:36:32.047964 | 2026-03-13 01:36:32.048093 | TASK [Load automation vars with template resolution] 2026-03-13 01:36:32.139139 | hotstack | ok 2026-03-13 01:36:32.152590 | 2026-03-13 01:36:32.152837 | TASK [Fetch stack outputs from remote to zuul executor] 2026-03-13 01:36:32.477406 | hotstack | Output suppressed because no_log was given 2026-03-13 01:36:32.487472 | 2026-03-13 01:36:32.487598 | TASK [Load stack output vars with template resolution] 2026-03-13 01:36:32.534829 | hotstack | Output suppressed because no_log was given 2026-03-13 01:36:32.545525 | 2026-03-13 01:36:32.545646 | TASK [Add Hotstack controller-0 to ansible inventory] 2026-03-13 01:36:32.593190 | hotstack | Output suppressed because no_log was given 2026-03-13 01:36:32.603845 | 2026-03-13 01:36:32.603972 | TASK [Wait for Hotstack controller-0 to be ready] 2026-03-13 01:36:38.253969 | hotstack -> 38.102.83.243 | Output suppressed because no_log was given 2026-03-13 01:36:38.266954 | 2026-03-13 01:36:38.267137 | TASK [Run scenario RHOSO deploy automation] 2026-03-13 01:36:38.295975 | hotstack | ok 2026-03-13 01:36:38.347806 | 2026-03-13 01:36:38.347997 | TASK [hotloop : Initialize retry metrics] 2026-03-13 01:36:38.385117 | hotstack -> 38.102.83.243 | ok 2026-03-13 01:36:38.393991 | 2026-03-13 01:36:38.394104 | TASK [hotloop : Assert config is defined] 2026-03-13 01:36:38.796421 | hotstack -> 38.102.83.243 | ok: All assertions passed 2026-03-13 01:36:38.808827 | 2026-03-13 01:36:38.808969 | TASK [hotloop : Load stages] 2026-03-13 01:36:39.468844 | hotstack -> 38.102.83.243 | changed 2026-03-13 01:36:39.488429 | 2026-03-13 01:36:39.488565 | LOOP [hotloop : Ensure directory exists] 2026-03-13 01:36:39.934735 | hotstack -> 38.102.83.243 | changed: "/home/zuul/manifests" 2026-03-13 01:36:39.959520 | 2026-03-13 01:36:39.959753 | TASK [hotloop : Create temporary hotloop work directory] 2026-03-13 01:36:40.497362 | hotstack -> 38.102.83.243 | changed 2026-03-13 01:36:40.511255 | 2026-03-13 01:36:40.511416 | TASK [hotloop : Create temporary directory for templates] 2026-03-13 01:36:40.772906 | hotstack -> localhost | changed 2026-03-13 01:36:40.778880 | 2026-03-13 01:36:40.778949 | TASK [hotloop : Remove target directory for clean sync (local connection)] 2026-03-13 01:36:40.883726 | hotstack | skipping: Conditional result was False 2026-03-13 01:36:40.898415 | 2026-03-13 01:36:40.898557 | TASK [hotloop : Copy work files (local connection)] 2026-03-13 01:36:40.984158 | hotstack | skipping: Conditional result was False 2026-03-13 01:36:40.995966 | 2026-03-13 01:36:40.996098 | TASK [hotloop : Sync work files (remote connection) - using synchronize push] 2026-03-13 01:36:41.861314 | hotstack -> 38.102.83.155 | changed: 2026-03-13 01:36:41.861519 | hotstack -> 38.102.83.155 | .d..tp..... ./ 2026-03-13 01:36:41.861553 | hotstack -> 38.102.83.155 | 38.102.83.155 | 38.102.83.155 | 38.102.83.155 | 38.102.83.155 | cd+++++++++ manifests/ 2026-03-13 01:36:41.861642 | hotstack -> 38.102.83.155 | cd+++++++++ manifests/control-plane/ 2026-03-13 01:36:41.861658 | hotstack -> 38.102.83.155 | 38.102.83.155 | 38.102.83.155 | cd+++++++++ manifests/control-plane/networking/ 2026-03-13 01:36:41.861732 | hotstack -> 38.102.83.155 | 38.102.83.155 | 38.102.83.155 | 38.102.83.155 | 38.102.83.155 | cd+++++++++ manifests/dataplane/ 2026-03-13 01:36:41.861809 | hotstack -> 38.102.83.155 | 38.102.83.155 | 38.102.83.155 | 38.102.83.155 | 38.102.83.155 | cd+++++++++ manifests/update/ 2026-03-13 01:36:41.861904 | hotstack -> 38.102.83.155 | 38.102.83.155 | 38.102.83.155 | 38.102.83.155 | cd+++++++++ test-operator/ 2026-03-13 01:36:41.861968 | hotstack -> 38.102.83.155 | 38.102.83.155 | localhost | ok: 2026-03-13 01:36:42.546301 | hotstack -> localhost | { 2026-03-13 01:36:42.546358 | hotstack -> localhost | "documentation": "Install the TopoLVM Container Storage Interface (CSI) driver on the OCP\ncluster.\n\nIt uses LVMS (Logical Volume Manager Storage) to dynamically provision\nlocal storage built from block devices on OCP nodes.\n\nAfter you have installed LVM Storage, you must create an LVMCluster custom resource (CR).\n", 2026-03-13 01:36:42.546403 | hotstack -> localhost | "manifest": "/var/lib/zuul/builds/46a5d1853f074c73b1d21511708ab398/untrusted/project_0/github.com/openstack-k8s-operators/hotstack/roles/hotloop/files/common/manifests/deps/olm-topolvm.yaml", 2026-03-13 01:36:42.546464 | hotstack -> localhost | "name": "TopoLVM common", 2026-03-13 01:36:42.546501 | hotstack -> localhost | "wait_conditions": [ 2026-03-13 01:36:42.546536 | hotstack -> localhost | "oc wait namespaces openshift-storage --for jsonpath='{.status.phase}'=Active --timeout=180s", 2026-03-13 01:36:42.546571 | hotstack -> localhost | "oc wait -n openshift-storage ClusterServiceVersion -l operators.coreos.com/lvms-operator.openshift-storage --for jsonpath='{.status.phase}'=Succeeded --timeout=300s" 2026-03-13 01:36:42.546606 | hotstack -> localhost | ] 2026-03-13 01:36:42.546641 | hotstack -> localhost | } 2026-03-13 01:36:42.546754 | hotstack -> localhost | changed: All items complete 2026-03-13 01:36:42.546801 | 2026-03-13 01:36:43.062994 | hotstack -> localhost | ok: 2026-03-13 01:36:43.063122 | hotstack -> localhost | { 2026-03-13 01:36:43.063145 | hotstack -> localhost | "documentation": "Install the OpenStack K8S dependencies. (Namespaces, OperatorGroup, and Subscription CRs.)\n\nOnce these CRs are created, run the `oc wait` commands to confirm that\neach step of this procedure is complete.\n", 2026-03-13 01:36:43.063163 | hotstack -> localhost | "manifest": "/var/lib/zuul/builds/46a5d1853f074c73b1d21511708ab398/untrusted/project_0/github.com/openstack-k8s-operators/hotstack/roles/hotloop/files/common/manifests/deps/olm-deps.yaml", 2026-03-13 01:36:43.063180 | hotstack -> localhost | "name": "Common OLM Dependencies", 2026-03-13 01:36:43.063196 | hotstack -> localhost | "wait_conditions": [ 2026-03-13 01:36:43.063212 | hotstack -> localhost | "oc wait namespaces cert-manager-operator --for jsonpath='{.status.phase}'=Active --timeout=30s", 2026-03-13 01:36:43.063227 | hotstack -> localhost | "oc wait namespaces metallb-system --for jsonpath='{.status.phase}'=Active --timeout=30s", 2026-03-13 01:36:43.063243 | hotstack -> localhost | "oc wait namespaces openshift-nmstate --for jsonpath='{.status.phase}'=Active --timeout=30s" 2026-03-13 01:36:43.063258 | hotstack -> localhost | ] 2026-03-13 01:36:43.063273 | hotstack -> localhost | } 2026-03-13 01:36:43.635946 | hotstack -> localhost | ok: 2026-03-13 01:36:43.636089 | hotstack -> localhost | { 2026-03-13 01:36:43.636112 | hotstack -> localhost | "documentation": "Wait for the ClusterServiceVersions (CSVs) to reach 'Succeeded' phase.\nThis ensures OLM has finished installing the operators and their deployments exist.\n", 2026-03-13 01:36:43.636131 | hotstack -> localhost | "name": "Wait for OLM dependency CSVs", 2026-03-13 01:36:43.636159 | hotstack -> localhost | "wait_conditions": [ 2026-03-13 01:36:43.636175 | hotstack -> localhost | "oc wait -n metallb-system csv -l operators.coreos.com/metallb-operator.metallb-system= --for jsonpath='{.status.phase}'=Succeeded --timeout=300s", 2026-03-13 01:36:43.636191 | hotstack -> localhost | "oc wait -n cert-manager-operator csv -l operators.coreos.com/openshift-cert-manager-operator.cert-manager-operator= --for jsonpath='{.status.phase}'=Succeeded --timeout=300s", 2026-03-13 01:36:43.636206 | hotstack -> localhost | "oc wait -n openshift-nmstate csv -l operators.coreos.com/kubernetes-nmstate-operator.openshift-nmstate= --for jsonpath='{.status.phase}'=Succeeded --timeout=300s" 2026-03-13 01:36:43.636222 | hotstack -> localhost | ] 2026-03-13 01:36:43.636237 | hotstack -> localhost | } 2026-03-13 01:36:44.086712 | hotstack -> localhost | ok: 2026-03-13 01:36:44.086885 | hotstack -> localhost | { 2026-03-13 01:36:44.086935 | hotstack -> localhost | "documentation": "Wait for operator deployments to become available.\nAt this point, the CSVs are succeeded and deployments should exist.\n", 2026-03-13 01:36:44.086975 | hotstack -> localhost | "name": "Wait for OLM dependency deployments", 2026-03-13 01:36:44.087011 | hotstack -> localhost | "wait_conditions": [ 2026-03-13 01:36:44.087047 | hotstack -> localhost | "oc wait -n metallb-system deployment -l control-plane=controller-manager --for condition=Available --timeout=180s", 2026-03-13 01:36:44.087082 | hotstack -> localhost | "oc wait -n metallb-system deployment -l component=webhook-server --for condition=Available --timeout=180s", 2026-03-13 01:36:44.087116 | hotstack -> localhost | "oc wait -n cert-manager-operator deployment cert-manager-operator-controller-manager --for condition=Available --timeout=180s", 2026-03-13 01:36:44.087152 | hotstack -> localhost | "oc wait -n cert-manager deployment -l app=cainjector --for condition=Available --timeout=180s", 2026-03-13 01:36:44.087188 | hotstack -> localhost | "oc wait -n cert-manager deployment -l app=webhook --for condition=Available --timeout=180s", 2026-03-13 01:36:44.087223 | hotstack -> localhost | "oc wait -n cert-manager deployment -l app=cert-manager --for condition=Available --timeout=180s" 2026-03-13 01:36:44.087293 | hotstack -> localhost | ] 2026-03-13 01:36:44.087329 | hotstack -> localhost | } 2026-03-13 01:36:44.550262 | hotstack -> localhost | ok: 2026-03-13 01:36:44.550392 | hotstack -> localhost | { 2026-03-13 01:36:44.550416 | hotstack -> localhost | "manifest": "/var/lib/zuul/builds/46a5d1853f074c73b1d21511708ab398/untrusted/project_0/github.com/openstack-k8s-operators/hotstack/roles/hotloop/files/common/manifests/deps/metallb.yaml", 2026-03-13 01:36:44.550434 | hotstack -> localhost | "name": "Common MetalLB" 2026-03-13 01:36:44.550451 | hotstack -> localhost | } 2026-03-13 01:36:45.030717 | hotstack -> localhost | ok: 2026-03-13 01:36:45.030844 | hotstack -> localhost | { 2026-03-13 01:36:45.030868 | hotstack -> localhost | "manifest": "/var/lib/zuul/builds/46a5d1853f074c73b1d21511708ab398/untrusted/project_0/github.com/openstack-k8s-operators/hotstack/roles/hotloop/files/common/manifests/deps/nmstate.yaml", 2026-03-13 01:36:45.030886 | hotstack -> localhost | "name": "Common NMState" 2026-03-13 01:36:45.030903 | hotstack -> localhost | } 2026-03-13 01:36:45.467105 | hotstack -> localhost | ok: 2026-03-13 01:36:45.467256 | hotstack -> localhost | { 2026-03-13 01:36:45.467288 | hotstack -> localhost | "name": "Wait for MetalLB and NMState", 2026-03-13 01:36:45.467312 | hotstack -> localhost | "wait_conditions": [ 2026-03-13 01:36:45.467335 | hotstack -> localhost | "oc wait -n metallb-system daemonset -l component=speaker --for jsonpath='{.status.numberReady}'=1 --timeout=180s", 2026-03-13 01:36:45.467357 | hotstack -> localhost | "oc wait -n openshift-nmstate daemonset -l component=kubernetes-nmstate-handler --for jsonpath='{.status.numberReady}'=1 --timeout=180s", 2026-03-13 01:36:45.467394 | hotstack -> localhost | "oc wait -n openshift-nmstate deployment -l component=kubernetes-nmstate-webhook --for condition=Available --timeout=180s" 2026-03-13 01:36:45.467416 | hotstack -> localhost | ] 2026-03-13 01:36:45.467437 | hotstack -> localhost | } 2026-03-13 01:36:45.977111 | hotstack -> localhost | ok: 2026-03-13 01:36:45.977284 | hotstack -> localhost | { 2026-03-13 01:36:45.977334 | hotstack -> localhost | "command": "oc label node master-0 openstack.org/cinder-lvm=", 2026-03-13 01:36:45.977374 | hotstack -> localhost | "documentation": "The LVM backend for Cinder is a special case as the storage data is on\nthe OpenShift node and has no external storage systems. This has several\nimplications.\n\n* To prevent issues with exported volumes, cinder-operator automatically\n uses the host network. The backend is configured to use the host's\n VLAN IP address. This means that the cinder-volume service doesn't\n need any networkAttachments.\n* There can only be one node with the label openstack.org/cinder-lvm=.\n Apply the label using the command: ``oc label node openstack.org/cinder-lvm=``\n", 2026-03-13 01:36:45.977413 | hotstack -> localhost | "name": "Node label cinder-lvm", 2026-03-13 01:36:45.977450 | hotstack -> localhost | "wait_conditions": [ 2026-03-13 01:36:45.977485 | hotstack -> localhost | "oc wait nodes -l openstack.org/cinder-lvm --for condition=Ready --timeout=30s" 2026-03-13 01:36:45.977551 | hotstack -> localhost | ] 2026-03-13 01:36:45.977590 | hotstack -> localhost | } 2026-03-13 01:36:46.441970 | hotstack -> localhost | ok: 2026-03-13 01:36:46.442150 | hotstack -> localhost | { 2026-03-13 01:36:46.442200 | hotstack -> localhost | "documentation": "Create a TopoLVM - LVMCluster on the Openshift cluster.\n\nConfigure the LVMCluster custom resource (CR) to perform the following\nactions:\n\n* Create LVM volume groups that you can use to provision persistent\n volume claims (PVCs).\n* Configure a list of devices that you want to add to the LVM volume\n groups.\n", 2026-03-13 01:36:46.442248 | hotstack -> localhost | "manifest": "/var/lib/zuul/builds/46a5d1853f074c73b1d21511708ab398/untrusted/project_0/github.com/openstack-k8s-operators/hotstack/roles/hotloop/files/common/manifests/topolvm/topolvmcluster.yaml", 2026-03-13 01:36:46.442286 | hotstack -> localhost | "name": "TopoLVM LVMCluster", 2026-03-13 01:36:46.442323 | hotstack -> localhost | "wait_conditions": [ 2026-03-13 01:36:46.442357 | hotstack -> localhost | "oc wait -n openshift-storage lvmcluster.lvm.topolvm.io/lvmcluster --for jsonpath='{.status.state}'=Ready --timeout=180s" 2026-03-13 01:36:46.442391 | hotstack -> localhost | ] 2026-03-13 01:36:46.442423 | hotstack -> localhost | } 2026-03-13 01:36:46.899605 | hotstack -> localhost | ok: 2026-03-13 01:36:46.899796 | hotstack -> localhost | { 2026-03-13 01:36:46.899846 | hotstack -> localhost | "documentation": "Install the OpenStack K8S operators (Namespaces, OperatorGroup, and Subscription CRs.)\n\nOnce these CRs are created, run the `oc wait` commands to confirm that\neach step of this procedure is complete.\n", 2026-03-13 01:36:46.899890 | hotstack -> localhost | "j2_manifest": "/var/lib/zuul/builds/46a5d1853f074c73b1d21511708ab398/untrusted/project_0/github.com/openstack-k8s-operators/hotstack/roles/hotloop/files/common/manifests/olm/olm-openstack.yaml.j2", 2026-03-13 01:36:46.899926 | hotstack -> localhost | "name": "Common Openstack OLM", 2026-03-13 01:36:46.899961 | hotstack -> localhost | "wait_conditions": [ 2026-03-13 01:36:46.899994 | hotstack -> localhost | "oc wait namespaces openstack-operators --for jsonpath='{.status.phase}'=Active --timeout=30s", 2026-03-13 01:36:46.900028 | hotstack -> localhost | "oc wait namespaces openstack --for jsonpath='{.status.phase}'=Active --timeout=30s", 2026-03-13 01:36:46.900062 | hotstack -> localhost | "oc -n openstack-operators wait subscriptions.operators.coreos.com openstack-operator --for jsonpath='{.status.installPlanRef.name}' --timeout=300s" 2026-03-13 01:36:46.900106 | hotstack -> localhost | ] 2026-03-13 01:36:46.900141 | hotstack -> localhost | } 2026-03-13 01:36:47.419182 | hotstack -> localhost | ok: 2026-03-13 01:36:47.419369 | hotstack -> localhost | { 2026-03-13 01:36:47.419457 | hotstack -> localhost | "command": "hotstack-approve-install-plan", 2026-03-13 01:36:47.419531 | hotstack -> localhost | "documentation": "Approve the OpenStack operator install plan created by the subscription.\nWaits for the ClusterServiceVersion to reach 'Succeeded' phase.\n", 2026-03-13 01:36:47.419602 | hotstack -> localhost | "name": "Approve openstack-operator Install plan", 2026-03-13 01:36:47.419712 | hotstack -> localhost | "wait_conditions": [ 2026-03-13 01:36:47.419794 | hotstack -> localhost | "oc wait -n openstack-operators csv -l operators.coreos.com/openstack-operator.openstack-operators= --for jsonpath='{.status.phase}'=Succeeded --timeout=300s" 2026-03-13 01:36:47.419861 | hotstack -> localhost | ] 2026-03-13 01:36:47.419926 | hotstack -> localhost | } 2026-03-13 01:36:47.897131 | hotstack -> localhost | ok: 2026-03-13 01:36:47.897375 | hotstack -> localhost | { 2026-03-13 01:36:47.897471 | hotstack -> localhost | "command": "hotstack-leader-election-tune", 2026-03-13 01:36:47.897562 | hotstack -> localhost | "documentation": "Apply performance tuning patches to OpenStack operator leader election settings.\n\nThis stage runs the hotstack-leader-election-tune command to optimize leader\nelection behavior for better stability. The tuning is conditionally applied based\non the OpenStack operators version, skipping versions v1.0.0-v1.0.11 where different\ntuning parameters are needed.\n", 2026-03-13 01:36:47.897726 | hotstack -> localhost | "name": "Patch openstack-operators leader election tuneables", 2026-03-13 01:36:47.897839 | hotstack -> localhost | "run_conditions": [ 2026-03-13 01:36:47.897916 | hotstack -> localhost | "True" 2026-03-13 01:36:47.897998 | hotstack -> localhost | ] 2026-03-13 01:36:47.898081 | hotstack -> localhost | } 2026-03-13 01:36:48.372138 | hotstack -> localhost | ok: 2026-03-13 01:36:48.372287 | hotstack -> localhost | { 2026-03-13 01:36:48.372326 | hotstack -> localhost | "documentation": "Create the primary OpenStack custom resource to initialize all operator components.\nWaits for the OpenStack resource and all associated deployments to become ready.\n", 2026-03-13 01:36:48.372358 | hotstack -> localhost | "manifest": "/var/lib/zuul/builds/46a5d1853f074c73b1d21511708ab398/untrusted/project_0/github.com/openstack-k8s-operators/hotstack/roles/hotloop/files/common/manifests/olm/openstack-init.yaml", 2026-03-13 01:36:48.372385 | hotstack -> localhost | "name": "Openstack operator initialization resource", 2026-03-13 01:36:48.372411 | hotstack -> localhost | "run_conditions": [ 2026-03-13 01:36:48.372443 | hotstack -> localhost | "True" 2026-03-13 01:36:48.372476 | hotstack -> localhost | ], 2026-03-13 01:36:48.372507 | hotstack -> localhost | "wait_conditions": [ 2026-03-13 01:36:48.372538 | hotstack -> localhost | "oc wait -n openstack-operators openstack openstack --for condition=Ready --timeout=300s", 2026-03-13 01:36:48.372567 | hotstack -> localhost | "oc wait -n openstack-operators -l openstack.org/operator-name deployment --for condition=Available --timeout=60s", 2026-03-13 01:36:48.372593 | hotstack -> localhost | "oc wait -n openstack-operators -l app.kubernetes.io/name=rabbitmq-cluster-operator deployment --for condition=Available --timeout=60s" 2026-03-13 01:36:48.372618 | hotstack -> localhost | ] 2026-03-13 01:36:48.372647 | hotstack -> localhost | } 2026-03-13 01:36:48.748455 | hotstack -> localhost | ok: 2026-03-13 01:36:48.748653 | hotstack -> localhost | { 2026-03-13 01:36:48.748802 | hotstack -> localhost | "manifest": "manifests/control-plane/networking/nncp.yaml", 2026-03-13 01:36:48.748877 | hotstack -> localhost | "name": "NodeNetworkConfigurationPolicy (nncp)", 2026-03-13 01:36:48.748942 | hotstack -> localhost | "wait_conditions": [ 2026-03-13 01:36:48.749020 | hotstack -> localhost | "oc wait -n openstack nncp -l osp/nncm-config-type=standard --for jsonpath='{.status.conditions[0].reason}'=SuccessfullyConfigured --timeout=180s" 2026-03-13 01:36:48.749082 | hotstack -> localhost | ] 2026-03-13 01:36:48.749149 | hotstack -> localhost | } 2026-03-13 01:36:49.209390 | hotstack -> localhost | ok: 2026-03-13 01:36:49.209536 | hotstack -> localhost | { 2026-03-13 01:36:49.209590 | hotstack -> localhost | "manifest": "manifests/control-plane/networking/nad.yaml", 2026-03-13 01:36:49.209632 | hotstack -> localhost | "name": "NetworkAttchmentDefinition (NAD)" 2026-03-13 01:36:49.209711 | hotstack -> localhost | } 2026-03-13 01:36:49.686394 | hotstack -> localhost | ok: 2026-03-13 01:36:49.686500 | hotstack -> localhost | { 2026-03-13 01:36:49.686523 | hotstack -> localhost | "manifest": "manifests/control-plane/networking/metallb.yaml", 2026-03-13 01:36:49.686541 | hotstack -> localhost | "name": "MetalLB - L2Advertisement and IPAddressPool" 2026-03-13 01:36:49.686558 | hotstack -> localhost | } 2026-03-13 01:36:50.177609 | hotstack -> localhost | ok: 2026-03-13 01:36:50.177741 | hotstack -> localhost | { 2026-03-13 01:36:50.177765 | hotstack -> localhost | "manifest": "manifests/control-plane/networking/netconfig.yaml", 2026-03-13 01:36:50.177783 | hotstack -> localhost | "name": "Netconfig" 2026-03-13 01:36:50.177799 | hotstack -> localhost | } 2026-03-13 01:36:50.626196 | hotstack -> localhost | ok: 2026-03-13 01:36:50.626342 | hotstack -> localhost | { 2026-03-13 01:36:50.626393 | hotstack -> localhost | "manifest": "manifests/control-plane/secrets.yaml", 2026-03-13 01:36:50.626435 | hotstack -> localhost | "name": "OpenstackControlPlane secrets" 2026-03-13 01:36:50.626474 | hotstack -> localhost | } 2026-03-13 01:36:51.090064 | hotstack -> localhost | ok: 2026-03-13 01:36:51.090209 | hotstack -> localhost | { 2026-03-13 01:36:51.090257 | hotstack -> localhost | "manifest": "manifests/control-plane/control-plane.yaml", 2026-03-13 01:36:51.090296 | hotstack -> localhost | "name": "Start OpenstackControlPlane deployment", 2026-03-13 01:36:51.090332 | hotstack -> localhost | "wait_conditions": [ 2026-03-13 01:36:51.090368 | hotstack -> localhost | "oc -n openstack wait openstackcontrolplanes.core.openstack.org controlplane --for condition=OpenStackControlPlaneDNSReadyCondition --timeout=600s", 2026-03-13 01:36:51.090404 | hotstack -> localhost | "oc -n openstack wait openstackcontrolplanes.core.openstack.org controlplane --for condition=OpenStackControlPlaneCAReadyCondition --timeout=600s" 2026-03-13 01:36:51.090438 | hotstack -> localhost | ] 2026-03-13 01:36:51.090472 | hotstack -> localhost | } 2026-03-13 01:36:51.584863 | hotstack -> localhost | ok: 2026-03-13 01:36:51.585015 | hotstack -> localhost | { 2026-03-13 01:36:51.585070 | hotstack -> localhost | "name": "Dataplane SSH key secret", 2026-03-13 01:36:51.585111 | hotstack -> localhost | "shell": "oc create -n openstack secret generic dataplane-ansible-ssh-private-key-secret --save-config --dry-run=client --from-file=ssh-privatekey=/home/zuul/.ssh/id_rsa --from-file=ssh-publickey=/home/zuul/.ssh/id_rsa.pub --type=Opaque -o yaml | oc apply -f -", 2026-03-13 01:36:51.585150 | hotstack -> localhost | "wait_conditions": [ 2026-03-13 01:36:51.585188 | hotstack -> localhost | "oc wait -n openstack secret dataplane-ansible-ssh-private-key-secret --for jsonpath='{.metadata.name}'=dataplane-ansible-ssh-private-key-secret --timeout=30s" 2026-03-13 01:36:51.585225 | hotstack -> localhost | ] 2026-03-13 01:36:51.585262 | hotstack -> localhost | } 2026-03-13 01:36:52.041852 | hotstack -> localhost | ok: 2026-03-13 01:36:52.041999 | hotstack -> localhost | { 2026-03-13 01:36:52.042047 | hotstack -> localhost | "name": "Nova migration SSH key secret", 2026-03-13 01:36:52.042086 | hotstack -> localhost | "shell": "oc create -n openstack secret generic nova-migration-ssh-key --save-config --dry-run=client --from-file=ssh-privatekey=/home/zuul/.ssh/id_nova_migrate --from-file=ssh-publickey=/home/zuul/.ssh/id_nova_migrate.pub --type=Opaque -o yaml | oc apply -f -", 2026-03-13 01:36:52.042135 | hotstack -> localhost | "wait_conditions": [ 2026-03-13 01:36:52.042173 | hotstack -> localhost | "oc wait -n openstack secret nova-migration-ssh-key --for jsonpath='{.metadata.name}'=nova-migration-ssh-key --timeout=30s" 2026-03-13 01:36:52.042209 | hotstack -> localhost | ] 2026-03-13 01:36:52.042244 | hotstack -> localhost | } 2026-03-13 01:36:52.510170 | hotstack -> localhost | ok: 2026-03-13 01:36:52.510316 | hotstack -> localhost | { 2026-03-13 01:36:52.510365 | hotstack -> localhost | "manifest": "manifests/dataplane/secrets.yaml", 2026-03-13 01:36:52.510404 | hotstack -> localhost | "name": "EDPM Secrets" 2026-03-13 01:36:52.510441 | hotstack -> localhost | } 2026-03-13 01:36:52.910236 | hotstack -> localhost | ok: 2026-03-13 01:36:52.910340 | hotstack -> localhost | { 2026-03-13 01:36:52.910365 | hotstack -> localhost | "manifest": "manifests/dataplane/nodeset.yaml", 2026-03-13 01:36:52.910383 | hotstack -> localhost | "name": "EDPM nodeset", 2026-03-13 01:36:52.910400 | hotstack -> localhost | "patches": [ 2026-03-13 01:36:52.910416 | hotstack -> localhost | { 2026-03-13 01:36:52.910431 | hotstack -> localhost | "path": "spec.nodeTemplate.ansible.ansibleVars.edpm_bootstrap_command", 2026-03-13 01:36:52.910447 | hotstack -> localhost | "value": "#!/bin/bash\n# Copyright Red Hat, Inc.\n# All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"); you may\n# not use this file except in compliance with the License. You may obtain\n# a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS, WITHOUT\n# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the\n# License for the specific language governing permissions and limitations\n# under the License.\n\nset -euxo pipefail\n\npushd /var/tmp\n\ncurl -sL https://github.com/openstack-k8s-operators/repo-setup/archive/refs/heads/main.tar.gz | tar -xz\n\npushd repo-setup-main\n\npython3 -m venv ./venv\nPBR_VERSION=0.0.0 ./venv/bin/pip install ./\n\n# This is required for FIPS enabled until trunk.rdoproject.org\n# is not being served from a centos7 host, tracked by\n# https://issues.redhat.com/browse/RHOSZUUL-1517\nupdate-crypto-policies --set FIPS:NO-ENFORCE-EMS\n\n./venv/bin/repo-setup current-podified -b antelope\n\npopd\n\nrm -rf repo-setup-main\n" 2026-03-13 01:36:52.910465 | hotstack -> localhost | }, 2026-03-13 01:36:52.910481 | hotstack -> localhost | { 2026-03-13 01:36:52.910496 | hotstack -> localhost | "path": "spec.nodeTemplate.ansible.ansibleVars.edpm_podman_registries", 2026-03-13 01:36:52.910512 | hotstack -> localhost | "value": [], 2026-03-13 01:36:52.910528 | hotstack -> localhost | "where": [ 2026-03-13 01:36:52.910543 | hotstack -> localhost | { 2026-03-13 01:36:52.910558 | hotstack -> localhost | "path": "spec.nodeTemplate.ansible.ansibleVars.edpm_podman_registries", 2026-03-13 01:36:52.910573 | hotstack -> localhost | "value": null 2026-03-13 01:36:52.910588 | hotstack -> localhost | } 2026-03-13 01:36:52.910603 | hotstack -> localhost | ] 2026-03-13 01:36:52.910618 | hotstack -> localhost | }, 2026-03-13 01:36:52.910633 | hotstack -> localhost | { 2026-03-13 01:36:52.910648 | hotstack -> localhost | "path": "spec.nodeTemplate.ansible.ansibleVars.edpm_container_registry_logins", 2026-03-13 01:36:52.910681 | hotstack -> localhost | "value": {}, 2026-03-13 01:36:52.910712 | hotstack -> localhost | "where": [ 2026-03-13 01:36:52.910731 | hotstack -> localhost | { 2026-03-13 01:36:52.910747 | hotstack -> localhost | "path": "spec.nodeTemplate.ansible.ansibleVars.edpm_podman_registries", 2026-03-13 01:36:52.910762 | hotstack -> localhost | "value": null 2026-03-13 01:36:52.910784 | hotstack -> localhost | } 2026-03-13 01:36:52.910800 | hotstack -> localhost | ] 2026-03-13 01:36:52.910815 | hotstack -> localhost | } 2026-03-13 01:36:52.910830 | hotstack -> localhost | ], 2026-03-13 01:36:52.910845 | hotstack -> localhost | "wait_conditions": [ 2026-03-13 01:36:52.910860 | hotstack -> localhost | "oc -n openstack wait openstackdataplanenodeset edpm --for condition=SetupReady --timeout=10m" 2026-03-13 01:36:52.910875 | hotstack -> localhost | ] 2026-03-13 01:36:52.910890 | hotstack -> localhost | } 2026-03-13 01:36:53.307470 | hotstack -> localhost | ok: 2026-03-13 01:36:53.307576 | hotstack -> localhost | { 2026-03-13 01:36:53.307601 | hotstack -> localhost | "manifest": "manifests/dataplane/deployment-os.yaml", 2026-03-13 01:36:53.307620 | hotstack -> localhost | "name": "EDPM deployment - OperatingSystem", 2026-03-13 01:36:53.307638 | hotstack -> localhost | "wait_conditions": [ 2026-03-13 01:36:53.307654 | hotstack -> localhost | "oc wait -n openstack jobs.batch bootstrap-dataplane-os-edpm --for condition=Complete --timeout=10m", 2026-03-13 01:36:53.307699 | hotstack -> localhost | "oc wait -n openstack jobs.batch configure-network-dataplane-os-edpm --for condition=Complete --timeout=5m", 2026-03-13 01:36:53.307723 | hotstack -> localhost | "oc wait -n openstack jobs.batch validate-network-dataplane-os-edpm --for condition=Complete --timeout=1m", 2026-03-13 01:36:53.307740 | hotstack -> localhost | "oc wait -n openstack jobs.batch install-os-dataplane-os-edpm --for condition=Complete --timeout=5m", 2026-03-13 01:36:53.307756 | hotstack -> localhost | "oc wait -n openstack jobs.batch configure-os-dataplane-os-edpm --for condition=Complete --timeout=3m", 2026-03-13 01:36:53.307771 | hotstack -> localhost | "oc wait -n openstack jobs.batch ssh-known-hosts-dataplane-os --for condition=Complete --timeout=1m", 2026-03-13 01:36:53.307788 | hotstack -> localhost | "oc wait -n openstack jobs.batch run-os-dataplane-os-edpm --for condition=Complete --timeout=3m", 2026-03-13 01:36:53.307804 | hotstack -> localhost | "oc wait -n openstack jobs.batch reboot-os-dataplane-os-edpm --for condition=Complete --timeout=5m", 2026-03-13 01:36:53.307819 | hotstack -> localhost | "oc wait -n openstack openstackdataplanedeployment dataplane-os --for condition=Ready --timeout=10m" 2026-03-13 01:36:53.307834 | hotstack -> localhost | ] 2026-03-13 01:36:53.307850 | hotstack -> localhost | } 2026-03-13 01:36:53.780022 | hotstack -> localhost | ok: 2026-03-13 01:36:53.780128 | hotstack -> localhost | { 2026-03-13 01:36:53.780152 | hotstack -> localhost | "manifest": "manifests/control-plane/control-plane.yaml", 2026-03-13 01:36:53.780171 | hotstack -> localhost | "name": "Wait for OpentackControlPlane Setup Ready", 2026-03-13 01:36:53.780188 | hotstack -> localhost | "wait_conditions": [ 2026-03-13 01:36:53.780204 | hotstack -> localhost | "oc wait -n openstack openstackcontrolplane controlplane --for condition=Ready --timeout=30m" 2026-03-13 01:36:53.780220 | hotstack -> localhost | ] 2026-03-13 01:36:53.780235 | hotstack -> localhost | } 2026-03-13 01:36:54.287228 | hotstack -> localhost | ok: 2026-03-13 01:36:54.287419 | hotstack -> localhost | { 2026-03-13 01:36:54.287496 | hotstack -> localhost | "manifest": "manifests/dataplane/deployment-services.yaml", 2026-03-13 01:36:54.287557 | hotstack -> localhost | "name": "EDPM deployment - Services", 2026-03-13 01:36:54.287614 | hotstack -> localhost | "wait_conditions": [ 2026-03-13 01:36:54.287709 | hotstack -> localhost | "oc wait -n openstack jobs.batch install-certs-dataplane-services-edpm --for condition=Complete --timeout=5m", 2026-03-13 01:36:54.287789 | hotstack -> localhost | "oc wait -n openstack jobs.batch ovn-dataplane-services-edpm --for condition=Complete --timeout=5m", 2026-03-13 01:36:54.287850 | hotstack -> localhost | "oc wait -n openstack jobs.batch neutron-metadata-dataplane-services-edpm --for condition=Complete --timeout=10m", 2026-03-13 01:36:54.287921 | hotstack -> localhost | "oc wait -n openstack jobs.batch libvirt-dataplane-services-edpm --for condition=Complete --timeout=20m", 2026-03-13 01:36:54.287979 | hotstack -> localhost | "oc wait -n openstack jobs.batch nova-dataplane-services-edpm --for condition=Complete --timeout=20m", 2026-03-13 01:36:54.288033 | hotstack -> localhost | "oc wait -n openstack jobs.batch telemetry-dataplane-services-edpm --for condition=Complete --timeout=10m", 2026-03-13 01:36:54.288089 | hotstack -> localhost | "oc wait -n openstack openstackdataplanedeployment dataplane-services --for condition=Ready --timeout=10m", 2026-03-13 01:36:54.288145 | hotstack -> localhost | "timeout --foreground 15m hotstack-nova-discover-hosts --namespace openstack --num-computes 1" 2026-03-13 01:36:54.288202 | hotstack -> localhost | ] 2026-03-13 01:36:54.288257 | hotstack -> localhost | } 2026-03-13 01:36:54.359541 | 2026-03-13 01:36:54.359629 | LOOP [hotloop : Execute automation stages] 2026-03-13 01:36:54.405633 | hotstack | ok: 2026-03-13 01:36:54.405906 | hotstack | { 2026-03-13 01:36:54.405965 | hotstack | "documentation": "Install the TopoLVM Container Storage Interface (CSI) driver on the OCP\ncluster.\n\nIt uses LVMS (Logical Volume Manager Storage) to dynamically provision\nlocal storage built from block devices on OCP nodes.\n\nAfter you have installed LVM Storage, you must create an LVMCluster custom resource (CR).\n", 2026-03-13 01:36:54.406020 | hotstack | "manifest": "/var/lib/zuul/builds/46a5d1853f074c73b1d21511708ab398/untrusted/project_0/github.com/openstack-k8s-operators/hotstack/roles/hotloop/files/common/manifests/deps/olm-topolvm.yaml", 2026-03-13 01:36:54.406070 | hotstack | "name": "TopoLVM common", 2026-03-13 01:36:54.406111 | hotstack | "wait_conditions": [ 2026-03-13 01:36:54.406150 | hotstack | "oc wait namespaces openshift-storage --for jsonpath='{.status.phase}'=Active --timeout=180s", 2026-03-13 01:36:54.406188 | hotstack | "oc wait -n openshift-storage ClusterServiceVersion -l operators.coreos.com/lvms-operator.openshift-storage --for jsonpath='{.status.phase}'=Succeeded --timeout=300s" 2026-03-13 01:36:54.406225 | hotstack | ] 2026-03-13 01:36:54.406263 | hotstack | } 2026-03-13 01:36:54.414497 | hotstack | ok: 2026-03-13 01:36:54.414559 | hotstack | { 2026-03-13 01:36:54.414590 | hotstack | "documentation": "Install the OpenStack K8S dependencies. (Namespaces, OperatorGroup, and Subscription CRs.)\n\nOnce these CRs are created, run the `oc wait` commands to confirm that\neach step of this procedure is complete.\n", 2026-03-13 01:36:54.414616 | hotstack | "manifest": "/var/lib/zuul/builds/46a5d1853f074c73b1d21511708ab398/untrusted/project_0/github.com/openstack-k8s-operators/hotstack/roles/hotloop/files/common/manifests/deps/olm-deps.yaml", 2026-03-13 01:36:54.414639 | hotstack | "name": "Common OLM Dependencies", 2026-03-13 01:36:54.414681 | hotstack | "wait_conditions": [ 2026-03-13 01:36:54.414712 | hotstack | "oc wait namespaces cert-manager-operator --for jsonpath='{.status.phase}'=Active --timeout=30s", 2026-03-13 01:36:54.414735 | hotstack | "oc wait namespaces metallb-system --for jsonpath='{.status.phase}'=Active --timeout=30s", 2026-03-13 01:36:54.414757 | hotstack | "oc wait namespaces openshift-nmstate --for jsonpath='{.status.phase}'=Active --timeout=30s" 2026-03-13 01:36:54.414779 | hotstack | ] 2026-03-13 01:36:54.414800 | hotstack | } 2026-03-13 01:36:54.422993 | hotstack | ok: 2026-03-13 01:36:54.423057 | hotstack | { 2026-03-13 01:36:54.423087 | hotstack | "documentation": "Wait for the ClusterServiceVersions (CSVs) to reach 'Succeeded' phase.\nThis ensures OLM has finished installing the operators and their deployments exist.\n", 2026-03-13 01:36:54.423112 | hotstack | "name": "Wait for OLM dependency CSVs", 2026-03-13 01:36:54.423135 | hotstack | "wait_conditions": [ 2026-03-13 01:36:54.423158 | hotstack | "oc wait -n metallb-system csv -l operators.coreos.com/metallb-operator.metallb-system= --for jsonpath='{.status.phase}'=Succeeded --timeout=300s", 2026-03-13 01:36:54.423189 | hotstack | "oc wait -n cert-manager-operator csv -l operators.coreos.com/openshift-cert-manager-operator.cert-manager-operator= --for jsonpath='{.status.phase}'=Succeeded --timeout=300s", 2026-03-13 01:36:54.423212 | hotstack | "oc wait -n openshift-nmstate csv -l operators.coreos.com/kubernetes-nmstate-operator.openshift-nmstate= --for jsonpath='{.status.phase}'=Succeeded --timeout=300s" 2026-03-13 01:36:54.423233 | hotstack | ] 2026-03-13 01:36:54.423255 | hotstack | } 2026-03-13 01:36:54.432219 | hotstack | ok: 2026-03-13 01:36:54.432281 | hotstack | { 2026-03-13 01:36:54.432312 | hotstack | "documentation": "Wait for operator deployments to become available.\nAt this point, the CSVs are succeeded and deployments should exist.\n", 2026-03-13 01:36:54.432337 | hotstack | "name": "Wait for OLM dependency deployments", 2026-03-13 01:36:54.432360 | hotstack | "wait_conditions": [ 2026-03-13 01:36:54.432382 | hotstack | "oc wait -n metallb-system deployment -l control-plane=controller-manager --for condition=Available --timeout=180s", 2026-03-13 01:36:54.432405 | hotstack | "oc wait -n metallb-system deployment -l component=webhook-server --for condition=Available --timeout=180s", 2026-03-13 01:36:54.432428 | hotstack | "oc wait -n cert-manager-operator deployment cert-manager-operator-controller-manager --for condition=Available --timeout=180s", 2026-03-13 01:36:54.432450 | hotstack | "oc wait -n cert-manager deployment -l app=cainjector --for condition=Available --timeout=180s", 2026-03-13 01:36:54.432472 | hotstack | "oc wait -n cert-manager deployment -l app=webhook --for condition=Available --timeout=180s", 2026-03-13 01:36:54.432493 | hotstack | "oc wait -n cert-manager deployment -l app=cert-manager --for condition=Available --timeout=180s" 2026-03-13 01:36:54.432514 | hotstack | ] 2026-03-13 01:36:54.432535 | hotstack | } 2026-03-13 01:36:54.441438 | hotstack | ok: 2026-03-13 01:36:54.441503 | hotstack | { 2026-03-13 01:36:54.441533 | hotstack | "manifest": "/var/lib/zuul/builds/46a5d1853f074c73b1d21511708ab398/untrusted/project_0/github.com/openstack-k8s-operators/hotstack/roles/hotloop/files/common/manifests/deps/metallb.yaml", 2026-03-13 01:36:54.441558 | hotstack | "name": "Common MetalLB" 2026-03-13 01:36:54.441582 | hotstack | } 2026-03-13 01:36:54.450583 | hotstack | ok: 2026-03-13 01:36:54.450658 | hotstack | { 2026-03-13 01:36:54.450743 | hotstack | "manifest": "/var/lib/zuul/builds/46a5d1853f074c73b1d21511708ab398/untrusted/project_0/github.com/openstack-k8s-operators/hotstack/roles/hotloop/files/common/manifests/deps/nmstate.yaml", 2026-03-13 01:36:54.450792 | hotstack | "name": "Common NMState" 2026-03-13 01:36:54.450835 | hotstack | } 2026-03-13 01:36:54.458180 | hotstack | ok: 2026-03-13 01:36:54.458242 | hotstack | { 2026-03-13 01:36:54.458274 | hotstack | "name": "Wait for MetalLB and NMState", 2026-03-13 01:36:54.458299 | hotstack | "wait_conditions": [ 2026-03-13 01:36:54.458322 | hotstack | "oc wait -n metallb-system daemonset -l component=speaker --for jsonpath='{.status.numberReady}'=1 --timeout=180s", 2026-03-13 01:36:54.458345 | hotstack | "oc wait -n openshift-nmstate daemonset -l component=kubernetes-nmstate-handler --for jsonpath='{.status.numberReady}'=1 --timeout=180s", 2026-03-13 01:36:54.458366 | hotstack | "oc wait -n openshift-nmstate deployment -l component=kubernetes-nmstate-webhook --for condition=Available --timeout=180s" 2026-03-13 01:36:54.458388 | hotstack | ] 2026-03-13 01:36:54.458410 | hotstack | } 2026-03-13 01:36:54.465646 | hotstack | ok: 2026-03-13 01:36:54.465727 | hotstack | { 2026-03-13 01:36:54.465759 | hotstack | "command": "oc label node master-0 openstack.org/cinder-lvm=", 2026-03-13 01:36:54.465784 | hotstack | "documentation": "The LVM backend for Cinder is a special case as the storage data is on\nthe OpenShift node and has no external storage systems. This has several\nimplications.\n\n* To prevent issues with exported volumes, cinder-operator automatically\n uses the host network. The backend is configured to use the host's\n VLAN IP address. This means that the cinder-volume service doesn't\n need any networkAttachments.\n* There can only be one node with the label openstack.org/cinder-lvm=.\n Apply the label using the command: ``oc label node openstack.org/cinder-lvm=``\n", 2026-03-13 01:36:54.465815 | hotstack | "name": "Node label cinder-lvm", 2026-03-13 01:36:54.465842 | hotstack | "wait_conditions": [ 2026-03-13 01:36:54.465875 | hotstack | "oc wait nodes -l openstack.org/cinder-lvm --for condition=Ready --timeout=30s" 2026-03-13 01:36:54.465907 | hotstack | ] 2026-03-13 01:36:54.465936 | hotstack | } 2026-03-13 01:36:54.473115 | hotstack | ok: 2026-03-13 01:36:54.473177 | hotstack | { 2026-03-13 01:36:54.473207 | hotstack | "documentation": "Create a TopoLVM - LVMCluster on the Openshift cluster.\n\nConfigure the LVMCluster custom resource (CR) to perform the following\nactions:\n\n* Create LVM volume groups that you can use to provision persistent\n volume claims (PVCs).\n* Configure a list of devices that you want to add to the LVM volume\n groups.\n", 2026-03-13 01:36:54.473231 | hotstack | "manifest": "/var/lib/zuul/builds/46a5d1853f074c73b1d21511708ab398/untrusted/project_0/github.com/openstack-k8s-operators/hotstack/roles/hotloop/files/common/manifests/topolvm/topolvmcluster.yaml", 2026-03-13 01:36:54.473255 | hotstack | "name": "TopoLVM LVMCluster", 2026-03-13 01:36:54.473277 | hotstack | "wait_conditions": [ 2026-03-13 01:36:54.473299 | hotstack | "oc wait -n openshift-storage lvmcluster.lvm.topolvm.io/lvmcluster --for jsonpath='{.status.state}'=Ready --timeout=180s" 2026-03-13 01:36:54.473320 | hotstack | ] 2026-03-13 01:36:54.473341 | hotstack | } 2026-03-13 01:36:54.480615 | hotstack | ok: 2026-03-13 01:36:54.480720 | hotstack | { 2026-03-13 01:36:54.480772 | hotstack | "documentation": "Install the OpenStack K8S operators (Namespaces, OperatorGroup, and Subscription CRs.)\n\nOnce these CRs are created, run the `oc wait` commands to confirm that\neach step of this procedure is complete.\n", 2026-03-13 01:36:54.480802 | hotstack | "j2_manifest": "/var/lib/zuul/builds/46a5d1853f074c73b1d21511708ab398/untrusted/project_0/github.com/openstack-k8s-operators/hotstack/roles/hotloop/files/common/manifests/olm/olm-openstack.yaml.j2", 2026-03-13 01:36:54.480826 | hotstack | "name": "Common Openstack OLM", 2026-03-13 01:36:54.480848 | hotstack | "wait_conditions": [ 2026-03-13 01:36:54.480870 | hotstack | "oc wait namespaces openstack-operators --for jsonpath='{.status.phase}'=Active --timeout=30s", 2026-03-13 01:36:54.480891 | hotstack | "oc wait namespaces openstack --for jsonpath='{.status.phase}'=Active --timeout=30s", 2026-03-13 01:36:54.480912 | hotstack | "oc -n openstack-operators wait subscriptions.operators.coreos.com openstack-operator --for jsonpath='{.status.installPlanRef.name}' --timeout=300s" 2026-03-13 01:36:54.480933 | hotstack | ] 2026-03-13 01:36:54.480954 | hotstack | } 2026-03-13 01:36:54.487547 | hotstack | ok: 2026-03-13 01:36:54.487592 | hotstack | { 2026-03-13 01:36:54.487614 | hotstack | "command": "hotstack-approve-install-plan", 2026-03-13 01:36:54.487632 | hotstack | "documentation": "Approve the OpenStack operator install plan created by the subscription.\nWaits for the ClusterServiceVersion to reach 'Succeeded' phase.\n", 2026-03-13 01:36:54.487649 | hotstack | "name": "Approve openstack-operator Install plan", 2026-03-13 01:36:54.487686 | hotstack | "wait_conditions": [ 2026-03-13 01:36:54.487714 | hotstack | "oc wait -n openstack-operators csv -l operators.coreos.com/openstack-operator.openstack-operators= --for jsonpath='{.status.phase}'=Succeeded --timeout=300s" 2026-03-13 01:36:54.487732 | hotstack | ] 2026-03-13 01:36:54.487747 | hotstack | } 2026-03-13 01:36:54.492716 | hotstack | ok: 2026-03-13 01:36:54.492773 | hotstack | { 2026-03-13 01:36:54.492796 | hotstack | "command": "hotstack-leader-election-tune", 2026-03-13 01:36:54.492819 | hotstack | "documentation": "Apply performance tuning patches to OpenStack operator leader election settings.\n\nThis stage runs the hotstack-leader-election-tune command to optimize leader\nelection behavior for better stability. The tuning is conditionally applied based\non the OpenStack operators version, skipping versions v1.0.0-v1.0.11 where different\ntuning parameters are needed.\n", 2026-03-13 01:36:54.492840 | hotstack | "name": "Patch openstack-operators leader election tuneables", 2026-03-13 01:36:54.492856 | hotstack | "run_conditions": [ 2026-03-13 01:36:54.492872 | hotstack | "True" 2026-03-13 01:36:54.492888 | hotstack | ] 2026-03-13 01:36:54.492903 | hotstack | } 2026-03-13 01:36:54.498099 | hotstack | ok: 2026-03-13 01:36:54.498144 | hotstack | { 2026-03-13 01:36:54.498166 | hotstack | "documentation": "Create the primary OpenStack custom resource to initialize all operator components.\nWaits for the OpenStack resource and all associated deployments to become ready.\n", 2026-03-13 01:36:54.498183 | hotstack | "manifest": "/var/lib/zuul/builds/46a5d1853f074c73b1d21511708ab398/untrusted/project_0/github.com/openstack-k8s-operators/hotstack/roles/hotloop/files/common/manifests/olm/openstack-init.yaml", 2026-03-13 01:36:54.498200 | hotstack | "name": "Openstack operator initialization resource", 2026-03-13 01:36:54.498216 | hotstack | "run_conditions": [ 2026-03-13 01:36:54.498232 | hotstack | "True" 2026-03-13 01:36:54.498247 | hotstack | ], 2026-03-13 01:36:54.498262 | hotstack | "wait_conditions": [ 2026-03-13 01:36:54.498278 | hotstack | "oc wait -n openstack-operators openstack openstack --for condition=Ready --timeout=300s", 2026-03-13 01:36:54.498292 | hotstack | "oc wait -n openstack-operators -l openstack.org/operator-name deployment --for condition=Available --timeout=60s", 2026-03-13 01:36:54.498307 | hotstack | "oc wait -n openstack-operators -l app.kubernetes.io/name=rabbitmq-cluster-operator deployment --for condition=Available --timeout=60s" 2026-03-13 01:36:54.498322 | hotstack | ] 2026-03-13 01:36:54.498337 | hotstack | } 2026-03-13 01:36:54.503302 | hotstack | ok: 2026-03-13 01:36:54.503346 | hotstack | { 2026-03-13 01:36:54.503368 | hotstack | "manifest": "manifests/control-plane/networking/nncp.yaml", 2026-03-13 01:36:54.503386 | hotstack | "name": "NodeNetworkConfigurationPolicy (nncp)", 2026-03-13 01:36:54.503403 | hotstack | "wait_conditions": [ 2026-03-13 01:36:54.503420 | hotstack | "oc wait -n openstack nncp -l osp/nncm-config-type=standard --for jsonpath='{.status.conditions[0].reason}'=SuccessfullyConfigured --timeout=180s" 2026-03-13 01:36:54.503436 | hotstack | ] 2026-03-13 01:36:54.503452 | hotstack | } 2026-03-13 01:36:54.508591 | hotstack | ok: 2026-03-13 01:36:54.508636 | hotstack | { 2026-03-13 01:36:54.508657 | hotstack | "manifest": "manifests/control-plane/networking/nad.yaml", 2026-03-13 01:36:54.508711 | hotstack | "name": "NetworkAttchmentDefinition (NAD)" 2026-03-13 01:36:54.508729 | hotstack | } 2026-03-13 01:36:54.513754 | hotstack | ok: 2026-03-13 01:36:54.513797 | hotstack | { 2026-03-13 01:36:54.513820 | hotstack | "manifest": "manifests/control-plane/networking/metallb.yaml", 2026-03-13 01:36:54.513838 | hotstack | "name": "MetalLB - L2Advertisement and IPAddressPool" 2026-03-13 01:36:54.513855 | hotstack | } 2026-03-13 01:36:54.518915 | hotstack | ok: 2026-03-13 01:36:54.518966 | hotstack | { 2026-03-13 01:36:54.518990 | hotstack | "manifest": "manifests/control-plane/networking/netconfig.yaml", 2026-03-13 01:36:54.519008 | hotstack | "name": "Netconfig" 2026-03-13 01:36:54.519024 | hotstack | } 2026-03-13 01:36:54.524019 | hotstack | ok: 2026-03-13 01:36:54.524071 | hotstack | { 2026-03-13 01:36:54.524093 | hotstack | "manifest": "manifests/control-plane/secrets.yaml", 2026-03-13 01:36:54.524112 | hotstack | "name": "OpenstackControlPlane secrets" 2026-03-13 01:36:54.524128 | hotstack | } 2026-03-13 01:36:54.529165 | hotstack | ok: 2026-03-13 01:36:54.529212 | hotstack | { 2026-03-13 01:36:54.529239 | hotstack | "manifest": "manifests/control-plane/control-plane.yaml", 2026-03-13 01:36:54.529257 | hotstack | "name": "Start OpenstackControlPlane deployment", 2026-03-13 01:36:54.529274 | hotstack | "wait_conditions": [ 2026-03-13 01:36:54.529290 | hotstack | "oc -n openstack wait openstackcontrolplanes.core.openstack.org controlplane --for condition=OpenStackControlPlaneDNSReadyCondition --timeout=600s", 2026-03-13 01:36:54.529306 | hotstack | "oc -n openstack wait openstackcontrolplanes.core.openstack.org controlplane --for condition=OpenStackControlPlaneCAReadyCondition --timeout=600s" 2026-03-13 01:36:54.529321 | hotstack | ] 2026-03-13 01:36:54.529337 | hotstack | } 2026-03-13 01:36:54.534344 | hotstack | ok: 2026-03-13 01:36:54.534387 | hotstack | { 2026-03-13 01:36:54.534408 | hotstack | "name": "Dataplane SSH key secret", 2026-03-13 01:36:54.534427 | hotstack | "shell": "oc create -n openstack secret generic dataplane-ansible-ssh-private-key-secret --save-config --dry-run=client --from-file=ssh-privatekey=/home/zuul/.ssh/id_rsa --from-file=ssh-publickey=/home/zuul/.ssh/id_rsa.pub --type=Opaque -o yaml | oc apply -f -", 2026-03-13 01:36:54.534444 | hotstack | "wait_conditions": [ 2026-03-13 01:36:54.534461 | hotstack | "oc wait -n openstack secret dataplane-ansible-ssh-private-key-secret --for jsonpath='{.metadata.name}'=dataplane-ansible-ssh-private-key-secret --timeout=30s" 2026-03-13 01:36:54.534479 | hotstack | ] 2026-03-13 01:36:54.534496 | hotstack | } 2026-03-13 01:36:54.539540 | hotstack | ok: 2026-03-13 01:36:54.539584 | hotstack | { 2026-03-13 01:36:54.539606 | hotstack | "name": "Nova migration SSH key secret", 2026-03-13 01:36:54.539624 | hotstack | "shell": "oc create -n openstack secret generic nova-migration-ssh-key --save-config --dry-run=client --from-file=ssh-privatekey=/home/zuul/.ssh/id_nova_migrate --from-file=ssh-publickey=/home/zuul/.ssh/id_nova_migrate.pub --type=Opaque -o yaml | oc apply -f -", 2026-03-13 01:36:54.539641 | hotstack | "wait_conditions": [ 2026-03-13 01:36:54.539657 | hotstack | "oc wait -n openstack secret nova-migration-ssh-key --for jsonpath='{.metadata.name}'=nova-migration-ssh-key --timeout=30s" 2026-03-13 01:36:54.539699 | hotstack | ] 2026-03-13 01:36:54.539725 | hotstack | } 2026-03-13 01:36:54.544815 | hotstack | ok: 2026-03-13 01:36:54.544860 | hotstack | { 2026-03-13 01:36:54.544881 | hotstack | "manifest": "manifests/dataplane/secrets.yaml", 2026-03-13 01:36:54.544899 | hotstack | "name": "EDPM Secrets" 2026-03-13 01:36:54.544916 | hotstack | } 2026-03-13 01:36:54.550164 | hotstack | ok: 2026-03-13 01:36:54.550208 | hotstack | { 2026-03-13 01:36:54.550230 | hotstack | "manifest": "manifests/dataplane/nodeset.yaml", 2026-03-13 01:36:54.550248 | hotstack | "name": "EDPM nodeset", 2026-03-13 01:36:54.550265 | hotstack | "patches": [ 2026-03-13 01:36:54.550281 | hotstack | { 2026-03-13 01:36:54.550296 | hotstack | "path": "spec.nodeTemplate.ansible.ansibleVars.edpm_bootstrap_command", 2026-03-13 01:36:54.550312 | hotstack | "value": "#!/bin/bash\n# Copyright Red Hat, Inc.\n# All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"); you may\n# not use this file except in compliance with the License. You may obtain\n# a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS, WITHOUT\n# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the\n# License for the specific language governing permissions and limitations\n# under the License.\n\nset -euxo pipefail\n\npushd /var/tmp\n\ncurl -sL https://github.com/openstack-k8s-operators/repo-setup/archive/refs/heads/main.tar.gz | tar -xz\n\npushd repo-setup-main\n\npython3 -m venv ./venv\nPBR_VERSION=0.0.0 ./venv/bin/pip install ./\n\n# This is required for FIPS enabled until trunk.rdoproject.org\n# is not being served from a centos7 host, tracked by\n# https://issues.redhat.com/browse/RHOSZUUL-1517\nupdate-crypto-policies --set FIPS:NO-ENFORCE-EMS\n\n./venv/bin/repo-setup current-podified -b antelope\n\npopd\n\nrm -rf repo-setup-main\n" 2026-03-13 01:36:54.550334 | hotstack | }, 2026-03-13 01:36:54.550351 | hotstack | { 2026-03-13 01:36:54.550366 | hotstack | "path": "spec.nodeTemplate.ansible.ansibleVars.edpm_podman_registries", 2026-03-13 01:36:54.550382 | hotstack | "value": [], 2026-03-13 01:36:54.550397 | hotstack | "where": [ 2026-03-13 01:36:54.550413 | hotstack | { 2026-03-13 01:36:54.550428 | hotstack | "path": "spec.nodeTemplate.ansible.ansibleVars.edpm_podman_registries", 2026-03-13 01:36:54.550443 | hotstack | "value": null 2026-03-13 01:36:54.550457 | hotstack | } 2026-03-13 01:36:54.550472 | hotstack | ] 2026-03-13 01:36:54.550487 | hotstack | }, 2026-03-13 01:36:54.550501 | hotstack | { 2026-03-13 01:36:54.550515 | hotstack | "path": "spec.nodeTemplate.ansible.ansibleVars.edpm_container_registry_logins", 2026-03-13 01:36:54.550530 | hotstack | "value": {}, 2026-03-13 01:36:54.550545 | hotstack | "where": [ 2026-03-13 01:36:54.550560 | hotstack | { 2026-03-13 01:36:54.550574 | hotstack | "path": "spec.nodeTemplate.ansible.ansibleVars.edpm_podman_registries", 2026-03-13 01:36:54.550589 | hotstack | "value": null 2026-03-13 01:36:54.550603 | hotstack | } 2026-03-13 01:36:54.550618 | hotstack | ] 2026-03-13 01:36:54.550635 | hotstack | } 2026-03-13 01:36:54.550650 | hotstack | ], 2026-03-13 01:36:54.550683 | hotstack | "wait_conditions": [ 2026-03-13 01:36:54.550701 | hotstack | "oc -n openstack wait openstackdataplanenodeset edpm --for condition=SetupReady --timeout=10m" 2026-03-13 01:36:54.550717 | hotstack | ] 2026-03-13 01:36:54.550732 | hotstack | } 2026-03-13 01:36:54.556040 | hotstack | ok: 2026-03-13 01:36:54.556086 | hotstack | { 2026-03-13 01:36:54.556107 | hotstack | "manifest": "manifests/dataplane/deployment-os.yaml", 2026-03-13 01:36:54.556125 | hotstack | "name": "EDPM deployment - OperatingSystem", 2026-03-13 01:36:54.556142 | hotstack | "wait_conditions": [ 2026-03-13 01:36:54.556162 | hotstack | "oc wait -n openstack jobs.batch bootstrap-dataplane-os-edpm --for condition=Complete --timeout=10m", 2026-03-13 01:36:54.556179 | hotstack | "oc wait -n openstack jobs.batch configure-network-dataplane-os-edpm --for condition=Complete --timeout=5m", 2026-03-13 01:36:54.556195 | hotstack | "oc wait -n openstack jobs.batch validate-network-dataplane-os-edpm --for condition=Complete --timeout=1m", 2026-03-13 01:36:54.556210 | hotstack | "oc wait -n openstack jobs.batch install-os-dataplane-os-edpm --for condition=Complete --timeout=5m", 2026-03-13 01:36:54.556226 | hotstack | "oc wait -n openstack jobs.batch configure-os-dataplane-os-edpm --for condition=Complete --timeout=3m", 2026-03-13 01:36:54.556240 | hotstack | "oc wait -n openstack jobs.batch ssh-known-hosts-dataplane-os --for condition=Complete --timeout=1m", 2026-03-13 01:36:54.556255 | hotstack | "oc wait -n openstack jobs.batch run-os-dataplane-os-edpm --for condition=Complete --timeout=3m", 2026-03-13 01:36:54.556270 | hotstack | "oc wait -n openstack jobs.batch reboot-os-dataplane-os-edpm --for condition=Complete --timeout=5m", 2026-03-13 01:36:54.556285 | hotstack | "oc wait -n openstack openstackdataplanedeployment dataplane-os --for condition=Ready --timeout=10m" 2026-03-13 01:36:54.556300 | hotstack | ] 2026-03-13 01:36:54.556315 | hotstack | } 2026-03-13 01:36:54.561379 | hotstack | ok: 2026-03-13 01:36:54.561424 | hotstack | { 2026-03-13 01:36:54.561445 | hotstack | "manifest": "manifests/control-plane/control-plane.yaml", 2026-03-13 01:36:54.561463 | hotstack | "name": "Wait for OpentackControlPlane Setup Ready", 2026-03-13 01:36:54.561479 | hotstack | "wait_conditions": [ 2026-03-13 01:36:54.561495 | hotstack | "oc wait -n openstack openstackcontrolplane controlplane --for condition=Ready --timeout=30m" 2026-03-13 01:36:54.561516 | hotstack | ] 2026-03-13 01:36:54.561533 | hotstack | } 2026-03-13 01:36:54.566777 | hotstack | ok: 2026-03-13 01:36:54.566823 | hotstack | { 2026-03-13 01:36:54.566845 | hotstack | "manifest": "manifests/dataplane/deployment-services.yaml", 2026-03-13 01:36:54.566863 | hotstack | "name": "EDPM deployment - Services", 2026-03-13 01:36:54.566880 | hotstack | "wait_conditions": [ 2026-03-13 01:36:54.566896 | hotstack | "oc wait -n openstack jobs.batch install-certs-dataplane-services-edpm --for condition=Complete --timeout=5m", 2026-03-13 01:36:54.566912 | hotstack | "oc wait -n openstack jobs.batch ovn-dataplane-services-edpm --for condition=Complete --timeout=5m", 2026-03-13 01:36:54.566927 | hotstack | "oc wait -n openstack jobs.batch neutron-metadata-dataplane-services-edpm --for condition=Complete --timeout=10m", 2026-03-13 01:36:54.566943 | hotstack | "oc wait -n openstack jobs.batch libvirt-dataplane-services-edpm --for condition=Complete --timeout=20m", 2026-03-13 01:36:54.566958 | hotstack | "oc wait -n openstack jobs.batch nova-dataplane-services-edpm --for condition=Complete --timeout=20m", 2026-03-13 01:36:54.566973 | hotstack | "oc wait -n openstack jobs.batch telemetry-dataplane-services-edpm --for condition=Complete --timeout=10m", 2026-03-13 01:36:54.566987 | hotstack | "oc wait -n openstack openstackdataplanedeployment dataplane-services --for condition=Ready --timeout=10m", 2026-03-13 01:36:54.567002 | hotstack | "timeout --foreground 15m hotstack-nova-discover-hosts --namespace openstack --num-computes 1" 2026-03-13 01:36:54.567017 | hotstack | ] 2026-03-13 01:36:54.567033 | hotstack | } 2026-03-13 01:36:54.607643 | hotstack | included: /var/lib/zuul/builds/46a5d1853f074c73b1d21511708ab398/work/tmp/ansible.1vc93szlhotloop_templates/stage_1.yml 2026-03-13 01:36:54.609283 | hotstack | included: /var/lib/zuul/builds/46a5d1853f074c73b1d21511708ab398/work/tmp/ansible.1vc93szlhotloop_templates/stage_2.yml 2026-03-13 01:36:54.610354 | hotstack | included: /var/lib/zuul/builds/46a5d1853f074c73b1d21511708ab398/work/tmp/ansible.1vc93szlhotloop_templates/stage_3.yml 2026-03-13 01:36:54.611437 | hotstack | included: /var/lib/zuul/builds/46a5d1853f074c73b1d21511708ab398/work/tmp/ansible.1vc93szlhotloop_templates/stage_4.yml 2026-03-13 01:36:54.612389 | hotstack | included: /var/lib/zuul/builds/46a5d1853f074c73b1d21511708ab398/work/tmp/ansible.1vc93szlhotloop_templates/stage_5.yml 2026-03-13 01:36:54.613352 | hotstack | included: /var/lib/zuul/builds/46a5d1853f074c73b1d21511708ab398/work/tmp/ansible.1vc93szlhotloop_templates/stage_6.yml 2026-03-13 01:36:54.614473 | hotstack | included: /var/lib/zuul/builds/46a5d1853f074c73b1d21511708ab398/work/tmp/ansible.1vc93szlhotloop_templates/stage_7.yml 2026-03-13 01:36:54.615803 | hotstack | included: /var/lib/zuul/builds/46a5d1853f074c73b1d21511708ab398/work/tmp/ansible.1vc93szlhotloop_templates/stage_8.yml 2026-03-13 01:36:54.617206 | hotstack | included: /var/lib/zuul/builds/46a5d1853f074c73b1d21511708ab398/work/tmp/ansible.1vc93szlhotloop_templates/stage_9.yml 2026-03-13 01:36:54.618736 | hotstack | included: /var/lib/zuul/builds/46a5d1853f074c73b1d21511708ab398/work/tmp/ansible.1vc93szlhotloop_templates/stage_10.yml 2026-03-13 01:36:54.620023 | hotstack | included: /var/lib/zuul/builds/46a5d1853f074c73b1d21511708ab398/work/tmp/ansible.1vc93szlhotloop_templates/stage_11.yml 2026-03-13 01:36:54.620900 | hotstack | included: /var/lib/zuul/builds/46a5d1853f074c73b1d21511708ab398/work/tmp/ansible.1vc93szlhotloop_templates/stage_12.yml 2026-03-13 01:36:54.622387 | hotstack | included: /var/lib/zuul/builds/46a5d1853f074c73b1d21511708ab398/work/tmp/ansible.1vc93szlhotloop_templates/stage_13.yml 2026-03-13 01:36:54.623786 | hotstack | included: /var/lib/zuul/builds/46a5d1853f074c73b1d21511708ab398/work/tmp/ansible.1vc93szlhotloop_templates/stage_14.yml 2026-03-13 01:36:54.624733 | hotstack | included: /var/lib/zuul/builds/46a5d1853f074c73b1d21511708ab398/work/tmp/ansible.1vc93szlhotloop_templates/stage_15.yml 2026-03-13 01:36:54.625690 | hotstack | included: /var/lib/zuul/builds/46a5d1853f074c73b1d21511708ab398/work/tmp/ansible.1vc93szlhotloop_templates/stage_16.yml 2026-03-13 01:36:54.626745 | hotstack | included: /var/lib/zuul/builds/46a5d1853f074c73b1d21511708ab398/work/tmp/ansible.1vc93szlhotloop_templates/stage_17.yml 2026-03-13 01:36:54.627635 | hotstack | included: /var/lib/zuul/builds/46a5d1853f074c73b1d21511708ab398/work/tmp/ansible.1vc93szlhotloop_templates/stage_18.yml 2026-03-13 01:36:54.629043 | hotstack | included: /var/lib/zuul/builds/46a5d1853f074c73b1d21511708ab398/work/tmp/ansible.1vc93szlhotloop_templates/stage_19.yml 2026-03-13 01:36:54.630320 | hotstack | included: /var/lib/zuul/builds/46a5d1853f074c73b1d21511708ab398/work/tmp/ansible.1vc93szlhotloop_templates/stage_20.yml 2026-03-13 01:36:54.631734 | hotstack | included: /var/lib/zuul/builds/46a5d1853f074c73b1d21511708ab398/work/tmp/ansible.1vc93szlhotloop_templates/stage_21.yml 2026-03-13 01:36:54.632659 | hotstack | included: /var/lib/zuul/builds/46a5d1853f074c73b1d21511708ab398/work/tmp/ansible.1vc93szlhotloop_templates/stage_22.yml 2026-03-13 01:36:54.634039 | hotstack | included: /var/lib/zuul/builds/46a5d1853f074c73b1d21511708ab398/work/tmp/ansible.1vc93szlhotloop_templates/stage_23.yml 2026-03-13 01:36:54.636016 | hotstack | included: /var/lib/zuul/builds/46a5d1853f074c73b1d21511708ab398/work/tmp/ansible.1vc93szlhotloop_templates/stage_24.yml 2026-03-13 01:36:54.637379 | hotstack | included: /var/lib/zuul/builds/46a5d1853f074c73b1d21511708ab398/work/tmp/ansible.1vc93szlhotloop_templates/stage_25.yml 2026-03-13 01:36:54.638834 | hotstack | included: /var/lib/zuul/builds/46a5d1853f074c73b1d21511708ab398/work/tmp/ansible.1vc93szlhotloop_templates/stage_26.yml 2026-03-13 01:36:54.671572 | 2026-03-13 01:36:54.671718 | TASK [hotloop : Stage: TopoLVM common :: Apply static manifest] 2026-03-13 01:36:54.702232 | hotstack | ok 2026-03-13 01:36:54.712160 | hotstack | included: /var/lib/zuul/builds/46a5d1853f074c73b1d21511708ab398/untrusted/project_0/github.com/openstack-k8s-operators/hotstack/roles/hotloop/tasks/static_manifest.yml 2026-03-13 01:36:54.718448 | 2026-03-13 01:36:54.718511 | TASK [hotloop : Stage: TopoLVM common :: Ensure manifest dir] 2026-03-13 01:36:55.089408 | hotstack -> 38.102.83.243 | changed 2026-03-13 01:36:55.104327 | 2026-03-13 01:36:55.104463 | TASK [hotloop : Stage: TopoLVM common :: Check if manifest exists in synced work directory] 2026-03-13 01:36:55.142816 | hotstack | skipping: Conditional result was False 2026-03-13 01:36:55.158959 | 2026-03-13 01:36:55.159104 | TASK [hotloop : Stage: TopoLVM common :: Copy manifest] 2026-03-13 01:36:55.861032 | hotstack -> 38.102.83.243 | changed 2026-03-13 01:36:55.876832 | 2026-03-13 01:36:55.876977 | LOOP [hotloop : Stage: TopoLVM common :: Apply patches] 2026-03-13 01:36:55.940371 | 2026-03-13 01:36:55.940725 | TASK [hotloop : Stage: TopoLVM common :: Apply static manifest] 2026-03-13 01:36:56.599302 | hotstack -> 38.102.83.243 | changed: Manifest file /home/zuul/manifests/deps/olm-topolvm.yaml applied and saved as /home/zuul/manifests/deps/olm-topolvm.yaml.applied with log at /home/zuul/manifests/deps/olm-topolvm.yaml.applied.log 2026-03-13 01:36:56.611735 | 2026-03-13 01:36:56.611869 | LOOP [hotloop : Stage: TopoLVM common :: Wait conditions] 2026-03-13 01:36:57.307721 | hotstack -> 38.102.83.243 | ok: "oc wait namespaces openshift-storage --for jsonpath='{.status.phase}'=Active --timeout=180s" 2026-03-13 01:36:57.308116 | 2026-03-13 01:37:19.139382 | hotstack -> 38.102.83.243 | ok: "oc wait -n openshift-storage ClusterServiceVersion -l operators.coreos.com/lvms-operator.openshift-storage --for jsonpath='{.status.phase}'=Succeeded --timeout=300s" 2026-03-13 01:37:19.156533 | 2026-03-13 01:37:19.156724 | TASK [hotloop : Stage: Common OLM Dependencies :: Apply static manifest] 2026-03-13 01:37:19.195101 | hotstack | ok 2026-03-13 01:37:19.214557 | hotstack | included: /var/lib/zuul/builds/46a5d1853f074c73b1d21511708ab398/untrusted/project_0/github.com/openstack-k8s-operators/hotstack/roles/hotloop/tasks/static_manifest.yml 2026-03-13 01:37:19.226243 | 2026-03-13 01:37:19.226334 | TASK [hotloop : Stage: Common OLM Dependencies :: Ensure manifest dir] 2026-03-13 01:37:19.597099 | hotstack -> 38.102.83.243 | ok 2026-03-13 01:37:19.612271 | 2026-03-13 01:37:19.612418 | TASK [hotloop : Stage: Common OLM Dependencies :: Check if manifest exists in synced work directory] 2026-03-13 01:37:19.650715 | hotstack | skipping: Conditional result was False 2026-03-13 01:37:19.665420 | 2026-03-13 01:37:19.665564 | TASK [hotloop : Stage: Common OLM Dependencies :: Copy manifest] 2026-03-13 01:37:20.345111 | hotstack -> 38.102.83.243 | changed 2026-03-13 01:37:20.351936 | 2026-03-13 01:37:20.352024 | LOOP [hotloop : Stage: Common OLM Dependencies :: Apply patches] 2026-03-13 01:37:20.392636 | 2026-03-13 01:37:20.392868 | TASK [hotloop : Stage: Common OLM Dependencies :: Apply static manifest] 2026-03-13 01:37:20.984786 | hotstack -> 38.102.83.243 | changed: Manifest file /home/zuul/manifests/deps/olm-deps.yaml applied and saved as /home/zuul/manifests/deps/olm-deps.yaml.applied with log at /home/zuul/manifests/deps/olm-deps.yaml.applied.log 2026-03-13 01:37:20.997809 | 2026-03-13 01:37:20.997945 | LOOP [hotloop : Stage: Common OLM Dependencies :: Wait conditions] 2026-03-13 01:37:21.580143 | hotstack -> 38.102.83.243 | ok: "oc wait namespaces cert-manager-operator --for jsonpath='{.status.phase}'=Active --timeout=30s" 2026-03-13 01:37:21.580631 | 2026-03-13 01:37:22.093182 | hotstack -> 38.102.83.243 | ok: "oc wait namespaces metallb-system --for jsonpath='{.status.phase}'=Active --timeout=30s" 2026-03-13 01:37:22.581945 | hotstack -> 38.102.83.243 | ok: "oc wait namespaces openshift-nmstate --for jsonpath='{.status.phase}'=Active --timeout=30s" 2026-03-13 01:37:22.600546 | 2026-03-13 01:37:22.600738 | LOOP [hotloop : Stage: Wait for OLM dependency CSVs :: Wait conditions] 2026-03-13 01:38:38.303537 | hotstack -> 38.102.83.243 | ok: "oc wait -n metallb-system csv -l operators.coreos.com/metallb-operator.metallb-system= --for jsonpath='{.status.phase}'=Succeeded --timeout=300s" 2026-03-13 01:38:38.811487 | hotstack -> 38.102.83.243 | ok: "oc wait -n cert-manager-operator csv -l operators.coreos.com/openshift-cert-manager-operator.cert-manager-operator= --for jsonpath='{.status.phase}'=Succeeded --timeout=300s" 2026-03-13 01:38:39.332899 | hotstack -> 38.102.83.243 | ok: "oc wait -n openshift-nmstate csv -l operators.coreos.com/kubernetes-nmstate-operator.openshift-nmstate= --for jsonpath='{.status.phase}'=Succeeded --timeout=300s" 2026-03-13 01:38:39.344762 | 2026-03-13 01:38:39.344938 | LOOP [hotloop : Stage: Wait for OLM dependency deployments :: Wait conditions] 2026-03-13 01:38:39.929388 | hotstack -> 38.102.83.243 | ok: "oc wait -n metallb-system deployment -l control-plane=controller-manager --for condition=Available --timeout=180s" 2026-03-13 01:38:39.929918 | hotstack -> 38.102.83.243 | ok: All items complete 2026-03-13 01:38:39.929982 | 2026-03-13 01:38:40.430191 | hotstack -> 38.102.83.243 | ok: "oc wait -n metallb-system deployment -l component=webhook-server --for condition=Available --timeout=180s" 2026-03-13 01:38:40.987477 | hotstack -> 38.102.83.243 | ok: "oc wait -n cert-manager-operator deployment cert-manager-operator-controller-manager --for condition=Available --timeout=180s" 2026-03-13 01:38:41.497000 | hotstack -> 38.102.83.243 | ok: "oc wait -n cert-manager deployment -l app=cainjector --for condition=Available --timeout=180s" 2026-03-13 01:38:41.980577 | hotstack -> 38.102.83.243 | ok: "oc wait -n cert-manager deployment -l app=webhook --for condition=Available --timeout=180s" 2026-03-13 01:38:42.498256 | hotstack -> 38.102.83.243 | ok: "oc wait -n cert-manager deployment -l app=cert-manager --for condition=Available --timeout=180s" 2026-03-13 01:38:42.517149 | 2026-03-13 01:38:42.517379 | TASK [hotloop : Stage: Common MetalLB :: Apply static manifest] 2026-03-13 01:38:42.556282 | hotstack | ok 2026-03-13 01:38:42.576173 | hotstack | included: /var/lib/zuul/builds/46a5d1853f074c73b1d21511708ab398/untrusted/project_0/github.com/openstack-k8s-operators/hotstack/roles/hotloop/tasks/static_manifest.yml 2026-03-13 01:38:42.588372 | 2026-03-13 01:38:42.588459 | TASK [hotloop : Stage: Common MetalLB :: Ensure manifest dir] 2026-03-13 01:38:42.973522 | hotstack -> 38.102.83.243 | ok 2026-03-13 01:38:42.988536 | 2026-03-13 01:38:42.988735 | TASK [hotloop : Stage: Common MetalLB :: Check if manifest exists in synced work directory] 2026-03-13 01:38:43.028405 | hotstack | skipping: Conditional result was False 2026-03-13 01:38:43.042364 | 2026-03-13 01:38:43.042507 | TASK [hotloop : Stage: Common MetalLB :: Copy manifest] 2026-03-13 01:38:43.733323 | hotstack -> 38.102.83.243 | changed 2026-03-13 01:38:43.744221 | 2026-03-13 01:38:43.744315 | LOOP [hotloop : Stage: Common MetalLB :: Apply patches] 2026-03-13 01:38:43.802091 | 2026-03-13 01:38:43.802321 | TASK [hotloop : Stage: Common MetalLB :: Apply static manifest] 2026-03-13 01:38:44.268502 | hotstack -> 38.102.83.243 | changed: Manifest file /home/zuul/manifests/deps/metallb.yaml applied and saved as /home/zuul/manifests/deps/metallb.yaml.applied with log at /home/zuul/manifests/deps/metallb.yaml.applied.log 2026-03-13 01:38:44.281995 | 2026-03-13 01:38:44.282130 | TASK [hotloop : Stage: Common NMState :: Apply static manifest] 2026-03-13 01:38:44.320003 | hotstack | ok 2026-03-13 01:38:44.340328 | hotstack | included: /var/lib/zuul/builds/46a5d1853f074c73b1d21511708ab398/untrusted/project_0/github.com/openstack-k8s-operators/hotstack/roles/hotloop/tasks/static_manifest.yml 2026-03-13 01:38:44.353280 | 2026-03-13 01:38:44.353372 | TASK [hotloop : Stage: Common NMState :: Ensure manifest dir] 2026-03-13 01:38:44.704654 | hotstack -> 38.102.83.243 | ok 2026-03-13 01:38:44.722599 | 2026-03-13 01:38:44.722864 | TASK [hotloop : Stage: Common NMState :: Check if manifest exists in synced work directory] 2026-03-13 01:38:44.760362 | hotstack | skipping: Conditional result was False 2026-03-13 01:38:44.776205 | 2026-03-13 01:38:44.776360 | TASK [hotloop : Stage: Common NMState :: Copy manifest] 2026-03-13 01:38:45.417243 | hotstack -> 38.102.83.243 | changed 2026-03-13 01:38:45.432495 | 2026-03-13 01:38:45.432646 | LOOP [hotloop : Stage: Common NMState :: Apply patches] 2026-03-13 01:38:45.494739 | 2026-03-13 01:38:45.495030 | TASK [hotloop : Stage: Common NMState :: Apply static manifest] 2026-03-13 01:38:45.929333 | hotstack -> 38.102.83.243 | changed: Manifest file /home/zuul/manifests/deps/nmstate.yaml applied and saved as /home/zuul/manifests/deps/nmstate.yaml.applied with log at /home/zuul/manifests/deps/nmstate.yaml.applied.log 2026-03-13 01:38:45.938246 | 2026-03-13 01:38:45.938333 | LOOP [hotloop : Stage: Wait for MetalLB and NMState :: Wait conditions] 2026-03-13 01:38:56.422472 | hotstack -> 38.102.83.243 | ok: "oc wait -n metallb-system daemonset -l component=speaker --for jsonpath='{.status.numberReady}'=1 --timeout=180s" 2026-03-13 01:38:56.422977 | 2026-03-13 01:39:02.154084 | hotstack -> 38.102.83.243 | ok: "oc wait -n openshift-nmstate daemonset -l component=kubernetes-nmstate-handler --for jsonpath='{.status.numberReady}'=1 --timeout=180s" 2026-03-13 01:39:07.602772 | hotstack -> 38.102.83.243 | ok: "oc wait -n openshift-nmstate deployment -l component=kubernetes-nmstate-webhook --for condition=Available --timeout=180s" 2026-03-13 01:39:07.621706 | 2026-03-13 01:39:07.621947 | TASK [hotloop : Stage: Node label cinder-lvm :: Run command] 2026-03-12 21:39:08.339038 | controller-0 | node/master-0 labeled 2026-03-13 01:39:08.656073 | hotstack -> 38.102.83.243 | changed 2026-03-13 01:39:08.669841 | 2026-03-13 01:39:08.669973 | LOOP [hotloop : Stage: Node label cinder-lvm :: Wait conditions] 2026-03-13 01:39:09.555239 | hotstack -> 38.102.83.243 | ok: "oc wait nodes -l openstack.org/cinder-lvm --for condition=Ready --timeout=30s" 2026-03-13 01:39:09.570864 | 2026-03-13 01:39:09.571170 | TASK [hotloop : Stage: TopoLVM LVMCluster :: Apply static manifest] 2026-03-13 01:39:09.612009 | hotstack | ok 2026-03-13 01:39:09.631744 | hotstack | included: /var/lib/zuul/builds/46a5d1853f074c73b1d21511708ab398/untrusted/project_0/github.com/openstack-k8s-operators/hotstack/roles/hotloop/tasks/static_manifest.yml 2026-03-13 01:39:09.645407 | 2026-03-13 01:39:09.645533 | TASK [hotloop : Stage: TopoLVM LVMCluster :: Ensure manifest dir] 2026-03-13 01:39:10.187186 | hotstack -> 38.102.83.243 | changed 2026-03-13 01:39:10.203538 | 2026-03-13 01:39:10.203711 | TASK [hotloop : Stage: TopoLVM LVMCluster :: Check if manifest exists in synced work directory] 2026-03-13 01:39:10.233514 | hotstack | skipping: Conditional result was False 2026-03-13 01:39:10.249297 | 2026-03-13 01:39:10.249449 | TASK [hotloop : Stage: TopoLVM LVMCluster :: Copy manifest] 2026-03-13 01:39:11.258980 | hotstack -> 38.102.83.243 | changed 2026-03-13 01:39:11.275088 | 2026-03-13 01:39:11.275241 | LOOP [hotloop : Stage: TopoLVM LVMCluster :: Apply patches] 2026-03-13 01:39:11.342190 | 2026-03-13 01:39:11.342471 | TASK [hotloop : Stage: TopoLVM LVMCluster :: Apply static manifest] 2026-03-13 01:39:12.126764 | hotstack -> 38.102.83.243 | changed: Manifest file /home/zuul/manifests/topolvm/topolvmcluster.yaml applied and saved as /home/zuul/manifests/topolvm/topolvmcluster.yaml.applied with log at /home/zuul/manifests/topolvm/topolvmcluster.yaml.applied.log 2026-03-13 01:39:12.140371 | 2026-03-13 01:39:12.140504 | LOOP [hotloop : Stage: TopoLVM LVMCluster :: Wait conditions] 2026-03-13 01:39:23.031505 | hotstack -> 38.102.83.243 | ok: "oc wait -n openshift-storage lvmcluster.lvm.topolvm.io/lvmcluster --for jsonpath='{.status.state}'=Ready --timeout=180s" 2026-03-13 01:39:23.031979 | 2026-03-13 01:39:23.050044 | 2026-03-13 01:39:23.050187 | TASK [hotloop : Stage: Common Openstack OLM :: Apply templates manifest] 2026-03-13 01:39:23.087509 | hotstack | ok 2026-03-13 01:39:23.108015 | hotstack | included: /var/lib/zuul/builds/46a5d1853f074c73b1d21511708ab398/untrusted/project_0/github.com/openstack-k8s-operators/hotstack/roles/hotloop/tasks/template_manifest.yml 2026-03-13 01:39:23.120749 | 2026-03-13 01:39:23.120838 | TASK [hotloop : Stage: Common Openstack OLM :: Ensure j2_manifest dir] 2026-03-13 01:39:23.471212 | hotstack -> 38.102.83.243 | changed 2026-03-13 01:39:23.488809 | 2026-03-13 01:39:23.488953 | TASK [hotloop : Stage: Common Openstack OLM :: Check if j2_manifest exists in synced work directory] 2026-03-13 01:39:23.527878 | hotstack | skipping: Conditional result was False 2026-03-13 01:39:23.542407 | 2026-03-13 01:39:23.542555 | TASK [hotloop : Stage: Common Openstack OLM :: Fetch template from remote when needed] 2026-03-13 01:39:23.611499 | hotstack | skipping: Conditional result was False 2026-03-13 01:39:23.626488 | 2026-03-13 01:39:23.626638 | TASK [hotloop : Stage: Common Openstack OLM :: Template manifest] 2026-03-13 01:39:24.349212 | hotstack -> 38.102.83.243 | changed 2026-03-13 01:39:24.364734 | 2026-03-13 01:39:24.364878 | LOOP [hotloop : Stage: Common Openstack OLM :: Apply patches] 2026-03-13 01:39:24.427954 | 2026-03-13 01:39:24.428200 | TASK [hotloop : Stage: Common Openstack OLM :: Apply static manifest] 2026-03-13 01:39:25.047443 | hotstack -> 38.102.83.243 | changed: Manifest file /home/zuul/manifests/olm/olm-openstack.yaml applied and saved as /home/zuul/manifests/olm/olm-openstack.yaml.applied with log at /home/zuul/manifests/olm/olm-openstack.yaml.applied.log 2026-03-13 01:39:25.060739 | 2026-03-13 01:39:25.060881 | LOOP [hotloop : Stage: Common Openstack OLM :: Wait conditions] 2026-03-13 01:39:25.647885 | hotstack -> 38.102.83.243 | ok: "oc wait namespaces openstack-operators --for jsonpath='{.status.phase}'=Active --timeout=30s" 2026-03-13 01:39:25.648521 | 2026-03-13 01:39:26.162469 | hotstack -> 38.102.83.243 | ok: "oc wait namespaces openstack --for jsonpath='{.status.phase}'=Active --timeout=30s" 2026-03-13 01:39:44.515257 | hotstack -> 38.102.83.243 | ok: "oc -n openstack-operators wait subscriptions.operators.coreos.com openstack-operator --for jsonpath='{.status.installPlanRef.name}' --timeout=300s" 2026-03-13 01:39:44.536630 | 2026-03-13 01:39:44.536902 | TASK [hotloop : Stage: Approve openstack-operator Install plan :: Run command] 2026-03-12 21:39:44.899108 | controller-0 | + '[' 1000 -eq 0 ']' 2026-03-12 21:39:44.899255 | controller-0 | + NAMESPACE=openstack-operators 2026-03-12 21:39:44.899270 | controller-0 | + SUBSCRIPTIONS=subscriptions.operators.coreos.com 2026-03-12 21:39:44.899277 | controller-0 | + INSTALL_PLANS=installplans.operators.coreos.com 2026-03-12 21:39:44.899284 | controller-0 | + CSV_REF=(-o jsonpath='{ .status.installedCSV }') 2026-03-12 21:39:44.899289 | controller-0 | + PLAN_REF=(-o jsonpath='{ .status.installPlanRef.name }') 2026-03-12 21:39:44.899293 | controller-0 | + SUBSCR_STATE=(-o jsonpath='{ .status.state }') 2026-03-12 21:39:44.899298 | controller-0 | + PATCH=(-p '{"spec":{"approved":true}}' --type merge) 2026-03-12 21:39:44.899302 | controller-0 | + NAME=openstack-operator 2026-03-12 21:39:44.899307 | controller-0 | + [[ 0 -gt 0 ]] 2026-03-12 21:39:44.899311 | controller-0 | + '[' -n '' ']' 2026-03-12 21:39:44.899316 | controller-0 | + approve_plan 2026-03-12 21:39:44.899323 | controller-0 | ++ oc -n openstack-operators get subscriptions.operators.coreos.com openstack-operator -o 'jsonpath={ .status.installedCSV }' 2026-03-12 21:39:45.053433 | controller-0 | + INSTALLED_CSV= 2026-03-12 21:39:45.053696 | controller-0 | + '[' -z '' ']' 2026-03-12 21:39:45.053711 | controller-0 | + patch_install_plan 2026-03-12 21:39:45.053719 | controller-0 | ++ oc -n openstack-operators get subscriptions.operators.coreos.com openstack-operator -o 'jsonpath={ .status.installPlanRef.name }' 2026-03-12 21:39:45.208043 | controller-0 | + PLAN=install-h7pp5 2026-03-12 21:39:45.376892 | controller-0 | + oc -n openstack-operators patch installplans.operators.coreos.com install-h7pp5 -p '{"spec":{"approved":true}}' --type merge 2026-03-12 21:39:45.376955 | controller-0 | installplan.operators.coreos.com/install-h7pp5 patched 2026-03-13 01:39:45.571391 | hotstack -> 38.102.83.243 | changed 2026-03-13 01:39:45.585364 | 2026-03-13 01:39:45.585497 | LOOP [hotloop : Stage: Approve openstack-operator Install plan :: Wait conditions] 2026-03-13 01:40:06.755392 | hotstack -> 38.102.83.243 | ok: "oc wait -n openstack-operators csv -l operators.coreos.com/openstack-operator.openstack-operators= --for jsonpath='{.status.phase}'=Succeeded --timeout=300s" 2026-03-13 01:40:06.776429 | 2026-03-13 01:40:06.776627 | TASK [hotloop : Stage: Patch openstack-operators leader election tuneables :: Run command] 2026-03-12 21:40:07.135850 | controller-0 | + '[' 1000 -eq 0 ']' 2026-03-12 21:40:07.136195 | controller-0 | ++ jq -r '.items[0].metadata.name' 2026-03-12 21:40:07.136385 | controller-0 | ++ oc -n openstack-operators get csv -l operators.coreos.com/openstack-operator.openstack-operators= -o json 2026-03-12 21:40:07.297167 | controller-0 | + CSV_NAME=openstack-operator.v0.6.0 2026-03-12 21:40:07.297505 | controller-0 | ++ jq '.spec.install.spec.deployments[].name | 2026-03-12 21:40:07.297527 | controller-0 | index("openstack-operator-controller-operator") | select( . != null )' 2026-03-12 21:40:07.298651 | controller-0 | ++ oc -n openstack-operators get csv openstack-operator.v0.6.0 -o json 2026-03-12 21:40:07.439505 | controller-0 | + DEPLOYMENT_INDEX= 2026-03-12 21:40:07.439666 | controller-0 | + '[' -z '' ']' 2026-03-12 21:40:07.439677 | controller-0 | + exit 0 2026-03-13 01:40:07.812631 | hotstack -> 38.102.83.243 | changed 2026-03-13 01:40:07.826751 | 2026-03-13 01:40:07.826891 | TASK [hotloop : Stage: Openstack operator initialization resource :: Apply static manifest] 2026-03-13 01:40:07.864214 | hotstack | ok 2026-03-13 01:40:07.883351 | hotstack | included: /var/lib/zuul/builds/46a5d1853f074c73b1d21511708ab398/untrusted/project_0/github.com/openstack-k8s-operators/hotstack/roles/hotloop/tasks/static_manifest.yml 2026-03-13 01:40:07.897055 | 2026-03-13 01:40:07.897144 | TASK [hotloop : Stage: Openstack operator initialization resource :: Ensure manifest dir] 2026-03-13 01:40:08.433923 | hotstack -> 38.102.83.243 | ok 2026-03-13 01:40:08.449031 | 2026-03-13 01:40:08.449175 | TASK [hotloop : Stage: Openstack operator initialization resource :: Check if manifest exists in synced work directory] 2026-03-13 01:40:08.487215 | hotstack | skipping: Conditional result was False 2026-03-13 01:40:08.501238 | 2026-03-13 01:40:08.501389 | TASK [hotloop : Stage: Openstack operator initialization resource :: Copy manifest] 2026-03-13 01:40:09.634456 | hotstack -> 38.102.83.243 | changed 2026-03-13 01:40:09.648477 | 2026-03-13 01:40:09.648616 | LOOP [hotloop : Stage: Openstack operator initialization resource :: Apply patches] 2026-03-13 01:40:09.691094 | 2026-03-13 01:40:09.691352 | TASK [hotloop : Stage: Openstack operator initialization resource :: Apply static manifest] 2026-03-13 01:40:10.472640 | hotstack -> 38.102.83.243 | changed: Manifest file /home/zuul/manifests/olm/openstack-init.yaml applied and saved as /home/zuul/manifests/olm/openstack-init.yaml.applied with log at /home/zuul/manifests/olm/openstack-init.yaml.applied.log 2026-03-13 01:40:10.486207 | 2026-03-13 01:40:10.486368 | LOOP [hotloop : Stage: Openstack operator initialization resource :: Wait conditions] 2026-03-13 01:41:12.813135 | hotstack -> 38.102.83.243 | ok: "oc wait -n openstack-operators openstack openstack --for condition=Ready --timeout=300s" 2026-03-13 01:41:12.813560 | 2026-03-13 01:41:15.887105 | hotstack -> 38.102.83.243 | ok: "oc wait -n openstack-operators -l openstack.org/operator-name deployment --for condition=Available --timeout=60s" 2026-03-13 01:41:16.363530 | hotstack -> 38.102.83.243 | ok: "oc wait -n openstack-operators -l app.kubernetes.io/name=rabbitmq-cluster-operator deployment --for condition=Available --timeout=60s" 2026-03-13 01:41:16.383830 | 2026-03-13 01:41:16.384013 | TASK [hotloop : Stage: NodeNetworkConfigurationPolicy (nncp) :: Apply static manifest] 2026-03-13 01:41:16.421425 | hotstack | ok 2026-03-13 01:41:16.440179 | hotstack | included: /var/lib/zuul/builds/46a5d1853f074c73b1d21511708ab398/untrusted/project_0/github.com/openstack-k8s-operators/hotstack/roles/hotloop/tasks/static_manifest.yml 2026-03-13 01:41:16.453351 | 2026-03-13 01:41:16.453438 | TASK [hotloop : Stage: NodeNetworkConfigurationPolicy (nncp) :: Ensure manifest dir] 2026-03-13 01:41:16.836727 | hotstack -> 38.102.83.243 | changed 2026-03-13 01:41:16.853216 | 2026-03-13 01:41:16.853356 | TASK [hotloop : Stage: NodeNetworkConfigurationPolicy (nncp) :: Check if manifest exists in synced work directory] 2026-03-13 01:41:17.220421 | hotstack -> 38.102.83.243 | ok 2026-03-13 01:41:17.236252 | 2026-03-13 01:41:17.236406 | TASK [hotloop : Stage: NodeNetworkConfigurationPolicy (nncp) :: Copy manifest] 2026-03-13 01:41:17.644343 | hotstack -> 38.102.83.243 | changed 2026-03-13 01:41:17.660144 | 2026-03-13 01:41:17.660285 | LOOP [hotloop : Stage: NodeNetworkConfigurationPolicy (nncp) :: Apply patches] 2026-03-13 01:41:17.722953 | 2026-03-13 01:41:17.723205 | TASK [hotloop : Stage: NodeNetworkConfigurationPolicy (nncp) :: Apply static manifest] 2026-03-13 01:41:18.355011 | hotstack -> 38.102.83.243 | changed: Manifest file /home/zuul/manifests/networking/nncp.yaml applied and saved as /home/zuul/manifests/networking/nncp.yaml.applied with log at /home/zuul/manifests/networking/nncp.yaml.applied.log 2026-03-13 01:41:18.370590 | 2026-03-13 01:41:18.370761 | LOOP [hotloop : Stage: NodeNetworkConfigurationPolicy (nncp) :: Wait conditions] 2026-03-13 01:41:39.603610 | hotstack -> 38.102.83.243 | ok: "oc wait -n openstack nncp -l osp/nncm-config-type=standard --for jsonpath='{.status.conditions[0].reason}'=SuccessfullyConfigured --timeout=180s" 2026-03-13 01:41:39.604060 | 2026-03-13 01:41:39.619473 | 2026-03-13 01:41:39.619626 | TASK [hotloop : Stage: NetworkAttchmentDefinition (NAD) :: Apply static manifest] 2026-03-13 01:41:39.657094 | hotstack | ok 2026-03-13 01:41:39.676641 | hotstack | included: /var/lib/zuul/builds/46a5d1853f074c73b1d21511708ab398/untrusted/project_0/github.com/openstack-k8s-operators/hotstack/roles/hotloop/tasks/static_manifest.yml 2026-03-13 01:41:39.688392 | 2026-03-13 01:41:39.688482 | TASK [hotloop : Stage: NetworkAttchmentDefinition (NAD) :: Ensure manifest dir] 2026-03-13 01:41:40.083394 | hotstack -> 38.102.83.243 | ok 2026-03-13 01:41:40.098917 | 2026-03-13 01:41:40.099063 | TASK [hotloop : Stage: NetworkAttchmentDefinition (NAD) :: Check if manifest exists in synced work directory] 2026-03-13 01:41:40.442873 | hotstack -> 38.102.83.243 | ok 2026-03-13 01:41:40.455944 | 2026-03-13 01:41:40.456043 | TASK [hotloop : Stage: NetworkAttchmentDefinition (NAD) :: Copy manifest] 2026-03-13 01:41:40.941220 | hotstack -> 38.102.83.243 | changed 2026-03-13 01:41:40.958808 | 2026-03-13 01:41:40.958955 | LOOP [hotloop : Stage: NetworkAttchmentDefinition (NAD) :: Apply patches] 2026-03-13 01:41:41.022551 | 2026-03-13 01:41:41.022925 | TASK [hotloop : Stage: NetworkAttchmentDefinition (NAD) :: Apply static manifest] 2026-03-13 01:41:41.602804 | hotstack -> 38.102.83.243 | changed: Manifest file /home/zuul/manifests/networking/nad.yaml applied and saved as /home/zuul/manifests/networking/nad.yaml.applied with log at /home/zuul/manifests/networking/nad.yaml.applied.log 2026-03-13 01:41:41.615993 | 2026-03-13 01:41:41.616130 | TASK [hotloop : Stage: MetalLB - L2Advertisement and IPAddressPool :: Apply static manifest] 2026-03-13 01:41:41.653477 | hotstack | ok 2026-03-13 01:41:41.672278 | hotstack | included: /var/lib/zuul/builds/46a5d1853f074c73b1d21511708ab398/untrusted/project_0/github.com/openstack-k8s-operators/hotstack/roles/hotloop/tasks/static_manifest.yml 2026-03-13 01:41:41.685528 | 2026-03-13 01:41:41.685619 | TASK [hotloop : Stage: MetalLB - L2Advertisement and IPAddressPool :: Ensure manifest dir] 2026-03-13 01:41:42.050649 | hotstack -> 38.102.83.243 | ok 2026-03-13 01:41:42.065534 | 2026-03-13 01:41:42.065707 | TASK [hotloop : Stage: MetalLB - L2Advertisement and IPAddressPool :: Check if manifest exists in synced work directory] 2026-03-13 01:41:42.429743 | hotstack -> 38.102.83.243 | ok 2026-03-13 01:41:42.446202 | 2026-03-13 01:41:42.446363 | TASK [hotloop : Stage: MetalLB - L2Advertisement and IPAddressPool :: Copy manifest] 2026-03-13 01:41:42.936147 | hotstack -> 38.102.83.243 | changed 2026-03-13 01:41:42.952140 | 2026-03-13 01:41:42.952287 | LOOP [hotloop : Stage: MetalLB - L2Advertisement and IPAddressPool :: Apply patches] 2026-03-13 01:41:43.015843 | 2026-03-13 01:41:43.016132 | TASK [hotloop : Stage: MetalLB - L2Advertisement and IPAddressPool :: Apply static manifest] 2026-03-13 01:41:43.831890 | hotstack -> 38.102.83.243 | changed: Manifest file /home/zuul/manifests/networking/metallb.yaml applied and saved as /home/zuul/manifests/networking/metallb.yaml.applied with log at /home/zuul/manifests/networking/metallb.yaml.applied.log 2026-03-13 01:41:43.844730 | 2026-03-13 01:41:43.844863 | TASK [hotloop : Stage: Netconfig :: Apply static manifest] 2026-03-13 01:41:43.882697 | hotstack | ok 2026-03-13 01:41:43.901633 | hotstack | included: /var/lib/zuul/builds/46a5d1853f074c73b1d21511708ab398/untrusted/project_0/github.com/openstack-k8s-operators/hotstack/roles/hotloop/tasks/static_manifest.yml 2026-03-13 01:41:43.914154 | 2026-03-13 01:41:43.914243 | TASK [hotloop : Stage: Netconfig :: Ensure manifest dir] 2026-03-13 01:41:44.334543 | hotstack -> 38.102.83.243 | ok 2026-03-13 01:41:44.351864 | 2026-03-13 01:41:44.352008 | TASK [hotloop : Stage: Netconfig :: Check if manifest exists in synced work directory] 2026-03-13 01:41:44.678577 | hotstack -> 38.102.83.243 | ok 2026-03-13 01:41:44.695157 | 2026-03-13 01:41:44.695305 | TASK [hotloop : Stage: Netconfig :: Copy manifest] 2026-03-13 01:41:45.118189 | hotstack -> 38.102.83.243 | changed 2026-03-13 01:41:45.132696 | 2026-03-13 01:41:45.132837 | LOOP [hotloop : Stage: Netconfig :: Apply patches] 2026-03-13 01:41:45.197021 | 2026-03-13 01:41:45.197272 | TASK [hotloop : Stage: Netconfig :: Apply static manifest] 2026-03-13 01:41:45.736625 | hotstack -> 38.102.83.243 | changed: Manifest file /home/zuul/manifests/networking/netconfig.yaml applied and saved as /home/zuul/manifests/networking/netconfig.yaml.applied with log at /home/zuul/manifests/networking/netconfig.yaml.applied.log 2026-03-13 01:41:45.752237 | 2026-03-13 01:41:45.752373 | TASK [hotloop : Stage: OpenstackControlPlane secrets :: Apply static manifest] 2026-03-13 01:41:45.844713 | hotstack | ok 2026-03-13 01:41:45.857140 | hotstack | included: /var/lib/zuul/builds/46a5d1853f074c73b1d21511708ab398/untrusted/project_0/github.com/openstack-k8s-operators/hotstack/roles/hotloop/tasks/static_manifest.yml 2026-03-13 01:41:45.862967 | 2026-03-13 01:41:45.863032 | TASK [hotloop : Stage: OpenstackControlPlane secrets :: Ensure manifest dir] 2026-03-13 01:41:46.257119 | hotstack -> 38.102.83.243 | changed 2026-03-13 01:41:46.272274 | 2026-03-13 01:41:46.272417 | TASK [hotloop : Stage: OpenstackControlPlane secrets :: Check if manifest exists in synced work directory] 2026-03-13 01:41:46.650115 | hotstack -> 38.102.83.243 | ok 2026-03-13 01:41:46.665582 | 2026-03-13 01:41:46.665770 | TASK [hotloop : Stage: OpenstackControlPlane secrets :: Copy manifest] 2026-03-13 01:41:47.123280 | hotstack -> 38.102.83.243 | changed 2026-03-13 01:41:47.138222 | 2026-03-13 01:41:47.138387 | LOOP [hotloop : Stage: OpenstackControlPlane secrets :: Apply patches] 2026-03-13 01:41:47.203421 | 2026-03-13 01:41:47.203702 | TASK [hotloop : Stage: OpenstackControlPlane secrets :: Apply static manifest] 2026-03-13 01:41:47.691976 | hotstack -> 38.102.83.243 | changed: Manifest file /home/zuul/manifests/control-plane/secrets.yaml applied and saved as /home/zuul/manifests/control-plane/secrets.yaml.applied with log at /home/zuul/manifests/control-plane/secrets.yaml.applied.log 2026-03-13 01:41:47.705554 | 2026-03-13 01:41:47.705717 | TASK [hotloop : Stage: Start OpenstackControlPlane deployment :: Apply static manifest] 2026-03-13 01:41:47.744501 | hotstack | ok 2026-03-13 01:41:47.764235 | hotstack | included: /var/lib/zuul/builds/46a5d1853f074c73b1d21511708ab398/untrusted/project_0/github.com/openstack-k8s-operators/hotstack/roles/hotloop/tasks/static_manifest.yml 2026-03-13 01:41:47.785116 | 2026-03-13 01:41:47.785404 | TASK [hotloop : Stage: Start OpenstackControlPlane deployment :: Ensure manifest dir] 2026-03-13 01:41:48.192835 | hotstack -> 38.102.83.243 | ok 2026-03-13 01:41:48.209561 | 2026-03-13 01:41:48.209761 | TASK [hotloop : Stage: Start OpenstackControlPlane deployment :: Check if manifest exists in synced work directory] 2026-03-13 01:41:48.566648 | hotstack -> 38.102.83.243 | ok 2026-03-13 01:41:48.577478 | 2026-03-13 01:41:48.577571 | TASK [hotloop : Stage: Start OpenstackControlPlane deployment :: Copy manifest] 2026-03-13 01:41:49.055748 | hotstack -> 38.102.83.243 | changed 2026-03-13 01:41:49.072537 | 2026-03-13 01:41:49.072723 | LOOP [hotloop : Stage: Start OpenstackControlPlane deployment :: Apply patches] 2026-03-13 01:41:49.140510 | 2026-03-13 01:41:49.140784 | TASK [hotloop : Stage: Start OpenstackControlPlane deployment :: Apply static manifest] 2026-03-13 01:41:49.817032 | hotstack -> 38.102.83.243 | changed: Manifest file /home/zuul/manifests/control-plane/control-plane.yaml applied and saved as /home/zuul/manifests/control-plane/control-plane.yaml.applied with log at /home/zuul/manifests/control-plane/control-plane.yaml.applied.log 2026-03-13 01:41:49.830800 | 2026-03-13 01:41:49.830940 | LOOP [hotloop : Stage: Start OpenstackControlPlane deployment :: Wait conditions] 2026-03-13 01:42:32.101733 | hotstack -> 38.102.83.243 | ok: "oc -n openstack wait openstackcontrolplanes.core.openstack.org controlplane --for condition=OpenStackControlPlaneDNSReadyCondition --timeout=600s" 2026-03-13 01:42:32.102195 | 2026-03-13 01:42:32.714157 | hotstack -> 38.102.83.243 | ok: "oc -n openstack wait openstackcontrolplanes.core.openstack.org controlplane --for condition=OpenStackControlPlaneCAReadyCondition --timeout=600s" 2026-03-13 01:42:32.729741 | 2026-03-13 01:42:32.729876 | TASK [hotloop : Stage: Dataplane SSH key secret :: Run shell] 2026-03-12 21:42:33.357125 | controller-0 | secret/dataplane-ansible-ssh-private-key-secret created 2026-03-13 01:42:33.766231 | hotstack -> 38.102.83.243 | changed 2026-03-13 01:42:33.779519 | 2026-03-13 01:42:33.779701 | LOOP [hotloop : Stage: Dataplane SSH key secret :: Wait conditions] 2026-03-13 01:42:34.958263 | hotstack -> 38.102.83.243 | ok: "oc wait -n openstack secret dataplane-ansible-ssh-private-key-secret --for jsonpath='{.metadata.name}'=dataplane-ansible-ssh-private-key-secret --timeout=30s" 2026-03-13 01:42:34.976340 | 2026-03-13 01:42:34.976474 | TASK [hotloop : Stage: Nova migration SSH key secret :: Run shell] 2026-03-12 21:42:35.990372 | controller-0 | secret/nova-migration-ssh-key created 2026-03-13 01:42:36.026761 | hotstack -> 38.102.83.243 | changed 2026-03-13 01:42:36.042054 | 2026-03-13 01:42:36.042181 | LOOP [hotloop : Stage: Nova migration SSH key secret :: Wait conditions] 2026-03-13 01:42:36.905869 | hotstack -> 38.102.83.243 | ok: "oc wait -n openstack secret nova-migration-ssh-key --for jsonpath='{.metadata.name}'=nova-migration-ssh-key --timeout=30s" 2026-03-13 01:42:36.906522 | hotstack -> 38.102.83.243 | ok: All items complete 2026-03-13 01:42:36.906593 | 2026-03-13 01:42:36.931415 | 2026-03-13 01:42:36.931623 | TASK [hotloop : Stage: EDPM Secrets :: Apply static manifest] 2026-03-13 01:42:36.969006 | hotstack | ok 2026-03-13 01:42:36.987904 | hotstack | included: /var/lib/zuul/builds/46a5d1853f074c73b1d21511708ab398/untrusted/project_0/github.com/openstack-k8s-operators/hotstack/roles/hotloop/tasks/static_manifest.yml 2026-03-13 01:42:36.999190 | 2026-03-13 01:42:36.999277 | TASK [hotloop : Stage: EDPM Secrets :: Ensure manifest dir] 2026-03-13 01:42:37.508213 | hotstack -> 38.102.83.243 | changed 2026-03-13 01:42:37.522511 | 2026-03-13 01:42:37.522706 | TASK [hotloop : Stage: EDPM Secrets :: Check if manifest exists in synced work directory] 2026-03-13 01:42:38.052000 | hotstack -> 38.102.83.243 | ok 2026-03-13 01:42:38.070196 | 2026-03-13 01:42:38.070364 | TASK [hotloop : Stage: EDPM Secrets :: Copy manifest] 2026-03-13 01:42:38.758938 | hotstack -> 38.102.83.243 | changed 2026-03-13 01:42:38.773692 | 2026-03-13 01:42:38.773837 | LOOP [hotloop : Stage: EDPM Secrets :: Apply patches] 2026-03-13 01:42:38.837164 | 2026-03-13 01:42:38.837402 | TASK [hotloop : Stage: EDPM Secrets :: Apply static manifest] 2026-03-13 01:42:39.599383 | hotstack -> 38.102.83.243 | changed: Manifest file /home/zuul/manifests/dataplane/secrets.yaml applied and saved as /home/zuul/manifests/dataplane/secrets.yaml.applied with log at /home/zuul/manifests/dataplane/secrets.yaml.applied.log 2026-03-13 01:42:39.612558 | 2026-03-13 01:42:39.612742 | TASK [hotloop : Stage: EDPM nodeset :: Apply static manifest] 2026-03-13 01:42:39.650039 | hotstack | ok 2026-03-13 01:42:39.668643 | hotstack | included: /var/lib/zuul/builds/46a5d1853f074c73b1d21511708ab398/untrusted/project_0/github.com/openstack-k8s-operators/hotstack/roles/hotloop/tasks/static_manifest.yml 2026-03-13 01:42:39.681186 | 2026-03-13 01:42:39.681272 | TASK [hotloop : Stage: EDPM nodeset :: Ensure manifest dir] 2026-03-13 01:42:40.182177 | hotstack -> 38.102.83.243 | ok 2026-03-13 01:42:40.191582 | 2026-03-13 01:42:40.191697 | TASK [hotloop : Stage: EDPM nodeset :: Check if manifest exists in synced work directory] 2026-03-13 01:42:40.704960 | hotstack -> 38.102.83.243 | ok 2026-03-13 01:42:40.720239 | 2026-03-13 01:42:40.720389 | TASK [hotloop : Stage: EDPM nodeset :: Copy manifest] 2026-03-13 01:42:41.398472 | hotstack -> 38.102.83.243 | changed 2026-03-13 01:42:41.413655 | 2026-03-13 01:42:41.413836 | LOOP [hotloop : Stage: EDPM nodeset :: Apply patches] 2026-03-13 01:42:42.118884 | hotstack -> 38.102.83.243 | changed: 2026-03-13 01:42:42.119229 | hotstack -> 38.102.83.243 | { 2026-03-13 01:42:42.119285 | hotstack -> 38.102.83.243 | "path": "spec.nodeTemplate.ansible.ansibleVars.edpm_bootstrap_command", 2026-03-13 01:42:42.119332 | hotstack -> 38.102.83.243 | "value": "#!/bin/bash\n# Copyright Red Hat, Inc.\n# All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"); you may\n# not use this file except in compliance with the License. You may obtain\n# a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS, WITHOUT\n# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the\n# License for the specific language governing permissions and limitations\n# under the License.\n\nset -euxo pipefail\n\npushd /var/tmp\n\ncurl -sL https://github.com/openstack-k8s-operators/repo-setup/archive/refs/heads/main.tar.gz | tar -xz\n\npushd repo-setup-main\n\npython3 -m venv ./venv\nPBR_VERSION=0.0.0 ./venv/bin/pip install ./\n\n# This is required for FIPS enabled until trunk.rdoproject.org\n# is not being served from a centos7 host, tracked by\n# https://issues.redhat.com/browse/RHOSZUUL-1517\nupdate-crypto-policies --set FIPS:NO-ENFORCE-EMS\n\n./venv/bin/repo-setup current-podified -b antelope\n\npopd\n\nrm -rf repo-setup-main\n" 2026-03-13 01:42:42.119385 | hotstack -> 38.102.83.243 | } 2026-03-13 01:42:42.119434 | 2026-03-13 01:42:42.641889 | hotstack -> 38.102.83.243 | changed: 2026-03-13 01:42:42.641998 | hotstack -> 38.102.83.243 | { 2026-03-13 01:42:42.642053 | hotstack -> 38.102.83.243 | "path": "spec.nodeTemplate.ansible.ansibleVars.edpm_podman_registries", 2026-03-13 01:42:42.642095 | hotstack -> 38.102.83.243 | "value": [], 2026-03-13 01:42:42.642132 | hotstack -> 38.102.83.243 | "where": [ 2026-03-13 01:42:42.642168 | hotstack -> 38.102.83.243 | { 2026-03-13 01:42:42.642203 | hotstack -> 38.102.83.243 | "path": "spec.nodeTemplate.ansible.ansibleVars.edpm_podman_registries", 2026-03-13 01:42:42.642236 | hotstack -> 38.102.83.243 | "value": null 2026-03-13 01:42:42.642269 | hotstack -> 38.102.83.243 | } 2026-03-13 01:42:42.642302 | hotstack -> 38.102.83.243 | ] 2026-03-13 01:42:42.642360 | hotstack -> 38.102.83.243 | } 2026-03-13 01:42:43.145615 | hotstack -> 38.102.83.243 | changed: 2026-03-13 01:42:43.145787 | hotstack -> 38.102.83.243 | { 2026-03-13 01:42:43.145839 | hotstack -> 38.102.83.243 | "path": "spec.nodeTemplate.ansible.ansibleVars.edpm_container_registry_logins", 2026-03-13 01:42:43.145879 | hotstack -> 38.102.83.243 | "value": {}, 2026-03-13 01:42:43.145914 | hotstack -> 38.102.83.243 | "where": [ 2026-03-13 01:42:43.145948 | hotstack -> 38.102.83.243 | { 2026-03-13 01:42:43.145981 | hotstack -> 38.102.83.243 | "path": "spec.nodeTemplate.ansible.ansibleVars.edpm_podman_registries", 2026-03-13 01:42:43.146014 | hotstack -> 38.102.83.243 | "value": null 2026-03-13 01:42:43.146047 | hotstack -> 38.102.83.243 | } 2026-03-13 01:42:43.146079 | hotstack -> 38.102.83.243 | ] 2026-03-13 01:42:43.146112 | hotstack -> 38.102.83.243 | } 2026-03-13 01:42:43.165127 | 2026-03-13 01:42:43.165326 | TASK [hotloop : Stage: EDPM nodeset :: Apply static manifest] 2026-03-13 01:42:44.111546 | hotstack -> 38.102.83.243 | changed: Manifest file /home/zuul/manifests/dataplane/nodeset.yaml applied and saved as /home/zuul/manifests/dataplane/nodeset.yaml.applied with log at /home/zuul/manifests/dataplane/nodeset.yaml.applied.log 2026-03-13 01:42:44.125287 | 2026-03-13 01:42:44.125417 | LOOP [hotloop : Stage: EDPM nodeset :: Wait conditions] 2026-03-13 01:42:50.648973 | hotstack -> 38.102.83.243 | ok: "oc -n openstack wait openstackdataplanenodeset edpm --for condition=SetupReady --timeout=10m" 2026-03-13 01:42:50.673289 | 2026-03-13 01:42:50.673478 | TASK [hotloop : Stage: EDPM deployment - OperatingSystem :: Apply static manifest] 2026-03-13 01:42:50.701840 | hotstack | ok 2026-03-13 01:42:50.721806 | hotstack | included: /var/lib/zuul/builds/46a5d1853f074c73b1d21511708ab398/untrusted/project_0/github.com/openstack-k8s-operators/hotstack/roles/hotloop/tasks/static_manifest.yml 2026-03-13 01:42:50.773116 | 2026-03-13 01:42:50.773249 | TASK [hotloop : Stage: EDPM deployment - OperatingSystem :: Ensure manifest dir] 2026-03-13 01:42:51.131342 | hotstack -> 38.102.83.243 | ok 2026-03-13 01:42:51.138496 | 2026-03-13 01:42:51.138562 | TASK [hotloop : Stage: EDPM deployment - OperatingSystem :: Check if manifest exists in synced work directory] 2026-03-13 01:42:51.427328 | hotstack -> 38.102.83.243 | ok 2026-03-13 01:42:51.434882 | 2026-03-13 01:42:51.434963 | TASK [hotloop : Stage: EDPM deployment - OperatingSystem :: Copy manifest] 2026-03-13 01:42:51.843327 | hotstack -> 38.102.83.243 | changed 2026-03-13 01:42:51.850555 | 2026-03-13 01:42:51.850637 | LOOP [hotloop : Stage: EDPM deployment - OperatingSystem :: Apply patches] 2026-03-13 01:42:51.902413 | 2026-03-13 01:42:51.902570 | TASK [hotloop : Stage: EDPM deployment - OperatingSystem :: Apply static manifest] 2026-03-13 01:42:52.395198 | hotstack -> 38.102.83.243 | changed: Manifest file /home/zuul/manifests/dataplane/deployment-os.yaml applied and saved as /home/zuul/manifests/dataplane/deployment-os.yaml.applied with log at /home/zuul/manifests/dataplane/deployment-os.yaml.applied.log 2026-03-13 01:42:52.408838 | 2026-03-13 01:42:52.408976 | LOOP [hotloop : Stage: EDPM deployment - OperatingSystem :: Wait conditions] 2026-03-13 01:47:49.645211 | hotstack -> 38.102.83.243 | ok: "oc wait -n openstack jobs.batch bootstrap-dataplane-os-edpm --for condition=Complete --timeout=10m" 2026-03-13 01:47:49.645604 | 2026-03-13 01:49:13.491364 | hotstack -> 38.102.83.243 | ok: "oc wait -n openstack jobs.batch configure-network-dataplane-os-edpm --for condition=Complete --timeout=5m" 2026-03-13 01:49:23.797253 | hotstack -> 38.102.83.243 | ok: "oc wait -n openstack jobs.batch validate-network-dataplane-os-edpm --for condition=Complete --timeout=1m" 2026-03-13 01:50:33.960356 | hotstack -> 38.102.83.243 | ok: "oc wait -n openstack jobs.batch install-os-dataplane-os-edpm --for condition=Complete --timeout=5m" 2026-03-13 01:51:40.635289 | hotstack -> 38.102.83.243 | ok: "oc wait -n openstack jobs.batch configure-os-dataplane-os-edpm --for condition=Complete --timeout=3m" 2026-03-13 01:51:56.699725 | hotstack -> 38.102.83.243 | ok: "oc wait -n openstack jobs.batch ssh-known-hosts-dataplane-os --for condition=Complete --timeout=1m" 2026-03-13 01:52:13.625869 | hotstack -> 38.102.83.243 | ok: "oc wait -n openstack jobs.batch run-os-dataplane-os-edpm --for condition=Complete --timeout=3m" 2026-03-13 01:53:36.418952 | hotstack -> 38.102.83.243 | ok: "oc wait -n openstack jobs.batch reboot-os-dataplane-os-edpm --for condition=Complete --timeout=5m" 2026-03-13 01:53:36.965337 | hotstack -> 38.102.83.243 | ok: "oc wait -n openstack openstackdataplanedeployment dataplane-os --for condition=Ready --timeout=10m" 2026-03-13 01:53:36.982813 | 2026-03-13 01:53:36.983035 | TASK [hotloop : Stage: Wait for OpentackControlPlane Setup Ready :: Apply static manifest] 2026-03-13 01:53:37.022203 | hotstack | ok 2026-03-13 01:53:37.041856 | hotstack | included: /var/lib/zuul/builds/46a5d1853f074c73b1d21511708ab398/untrusted/project_0/github.com/openstack-k8s-operators/hotstack/roles/hotloop/tasks/static_manifest.yml 2026-03-13 01:53:37.055296 | 2026-03-13 01:53:37.055381 | TASK [hotloop : Stage: Wait for OpentackControlPlane Setup Ready :: Ensure manifest dir] 2026-03-13 01:53:37.375841 | hotstack -> 38.102.83.243 | ok 2026-03-13 01:53:37.385720 | 2026-03-13 01:53:37.385811 | TASK [hotloop : Stage: Wait for OpentackControlPlane Setup Ready :: Check if manifest exists in synced work directory] 2026-03-13 01:53:37.747796 | hotstack -> 38.102.83.243 | ok 2026-03-13 01:53:37.762989 | 2026-03-13 01:53:37.763131 | TASK [hotloop : Stage: Wait for OpentackControlPlane Setup Ready :: Copy manifest] 2026-03-13 01:53:38.244997 | hotstack -> 38.102.83.243 | changed 2026-03-13 01:53:38.259581 | 2026-03-13 01:53:38.259759 | LOOP [hotloop : Stage: Wait for OpentackControlPlane Setup Ready :: Apply patches] 2026-03-13 01:53:38.323197 | 2026-03-13 01:53:38.323467 | TASK [hotloop : Stage: Wait for OpentackControlPlane Setup Ready :: Apply static manifest] 2026-03-13 01:53:38.680020 | hotstack -> 38.102.83.243 | ok: Manifest /home/zuul/manifests/control-plane/control-plane.yaml is not different from previously applied version /home/zuul/manifests/control-plane/control-plane.yaml.applied. No changes needed 2026-03-13 01:53:38.693143 | 2026-03-13 01:53:38.693280 | LOOP [hotloop : Stage: Wait for OpentackControlPlane Setup Ready :: Wait conditions] 2026-03-13 01:53:42.573857 | hotstack -> 38.102.83.243 | ok: "oc wait -n openstack openstackcontrolplane controlplane --for condition=Ready --timeout=30m" 2026-03-13 01:53:42.574127 | 2026-03-13 01:53:42.588962 | 2026-03-13 01:53:42.589161 | TASK [hotloop : Stage: EDPM deployment - Services :: Apply static manifest] 2026-03-13 01:53:42.628411 | hotstack | ok 2026-03-13 01:53:42.647605 | hotstack | included: /var/lib/zuul/builds/46a5d1853f074c73b1d21511708ab398/untrusted/project_0/github.com/openstack-k8s-operators/hotstack/roles/hotloop/tasks/static_manifest.yml 2026-03-13 01:53:42.660225 | 2026-03-13 01:53:42.660312 | TASK [hotloop : Stage: EDPM deployment - Services :: Ensure manifest dir] 2026-03-13 01:53:42.982233 | hotstack -> 38.102.83.243 | ok 2026-03-13 01:53:42.997001 | 2026-03-13 01:53:42.997145 | TASK [hotloop : Stage: EDPM deployment - Services :: Check if manifest exists in synced work directory] 2026-03-13 01:53:43.311949 | hotstack -> 38.102.83.243 | ok 2026-03-13 01:53:43.326558 | 2026-03-13 01:53:43.326736 | TASK [hotloop : Stage: EDPM deployment - Services :: Copy manifest] 2026-03-13 01:53:43.774954 | hotstack -> 38.102.83.243 | changed 2026-03-13 01:53:43.791960 | 2026-03-13 01:53:43.792103 | LOOP [hotloop : Stage: EDPM deployment - Services :: Apply patches] 2026-03-13 01:53:43.855385 | 2026-03-13 01:53:43.855633 | TASK [hotloop : Stage: EDPM deployment - Services :: Apply static manifest] 2026-03-13 01:53:44.335947 | hotstack -> 38.102.83.243 | changed: Manifest file /home/zuul/manifests/dataplane/deployment-services.yaml applied and saved as /home/zuul/manifests/dataplane/deployment-services.yaml.applied with log at /home/zuul/manifests/dataplane/deployment-services.yaml.applied.log 2026-03-13 01:53:44.349266 | 2026-03-13 01:53:44.349393 | LOOP [hotloop : Stage: EDPM deployment - Services :: Wait conditions] 2026-03-13 01:54:32.492468 | hotstack -> 38.102.83.243 | ok: "oc wait -n openstack jobs.batch install-certs-dataplane-services-edpm --for condition=Complete --timeout=5m" 2026-03-13 01:54:32.492868 | 2026-03-13 01:56:00.018008 | hotstack -> 38.102.83.243 | ok: "oc wait -n openstack jobs.batch ovn-dataplane-services-edpm --for condition=Complete --timeout=5m" 2026-03-13 01:57:12.356237 | hotstack -> 38.102.83.243 | ok: "oc wait -n openstack jobs.batch neutron-metadata-dataplane-services-edpm --for condition=Complete --timeout=10m" 2026-03-13 02:01:13.710089 | hotstack -> 38.102.83.243 | ok: "oc wait -n openstack jobs.batch libvirt-dataplane-services-edpm --for condition=Complete --timeout=20m" 2026-03-13 02:04:17.917598 | hotstack -> 38.102.83.243 | ok: "oc wait -n openstack jobs.batch nova-dataplane-services-edpm --for condition=Complete --timeout=20m" 2026-03-13 02:07:17.961824 | hotstack -> 38.102.83.243 | ok: "oc wait -n openstack jobs.batch telemetry-dataplane-services-edpm --for condition=Complete --timeout=10m" 2026-03-13 02:07:18.505104 | hotstack -> 38.102.83.243 | ok: "oc wait -n openstack openstackdataplanedeployment dataplane-services --for condition=Ready --timeout=10m" 2026-03-13 02:07:30.709566 | hotstack -> 38.102.83.243 | ok: "timeout --foreground 15m hotstack-nova-discover-hosts --namespace openstack --num-computes 1" 2026-03-13 02:07:30.733061 | 2026-03-13 02:07:30.733205 | TASK [hotloop : Display retry metrics summary] 2026-03-13 02:07:30.770597 | hotstack | ok 2026-03-13 02:07:30.783615 | hotstack | included: /var/lib/zuul/builds/46a5d1853f074c73b1d21511708ab398/untrusted/project_0/github.com/openstack-k8s-operators/hotstack/roles/hotloop/tasks/retry_metrics.yml 2026-03-13 02:07:30.825374 | 2026-03-13 02:07:30.825460 | TASK [hotloop : Display retry metrics summary] 2026-03-13 02:07:30.912511 | hotstack -> 38.102.83.243 | ok: 2026-03-13 02:07:30.912848 | hotstack -> 38.102.83.243 | 2026-03-13 02:07:30.912935 | hotstack -> 38.102.83.243 | ╔══════════════════════════════════════════════════════════════════╗ 2026-03-13 02:07:30.912989 | hotstack -> 38.102.83.243 | ║ HOTLOOP RETRY METRICS SUMMARY ║ 2026-03-13 02:07:30.913033 | hotstack -> 38.102.83.243 | ╚══════════════════════════════════════════════════════════════════╝ 2026-03-13 02:07:30.913074 | hotstack -> 38.102.83.243 | 2026-03-13 02:07:30.913114 | hotstack -> 38.102.83.243 | Total Stages with Retries: 0 2026-03-13 02:07:30.913153 | hotstack -> 38.102.83.243 | Total Retry Attempts: 0 2026-03-13 02:07:30.913192 | hotstack -> 38.102.83.243 | Total Retry Time: 0s 2026-03-13 02:07:30.925467 | 2026-03-13 02:07:30.925591 | TASK [hotloop : Remove temporary hotloop work directory] 2026-03-13 02:07:31.270344 | hotstack -> 38.102.83.243 | changed 2026-03-13 02:07:31.284213 | 2026-03-13 02:07:31.284357 | TASK [hotloop : Remove temporary templates directory] 2026-03-13 02:07:31.569627 | hotstack -> localhost | changed 2026-03-13 02:07:31.620024 | 2026-03-13 02:07:31.620134 | PLAY RECAP 2026-03-13 02:07:31.620208 | hotstack | ok: 153 changed: 55 unreachable: 0 failed: 0 skipped: 27 rescued: 0 ignored: 0 2026-03-13 02:07:31.620239 | 2026-03-13 02:07:31.786547 | RUN END RESULT_NORMAL: [untrusted : github.com/openstack-k8s-operators/hotstack/ci/playbooks/run-deploy.yml@main] 2026-03-13 02:07:31.791752 | RUN START: [untrusted : github.com/openstack-k8s-operators/hotstack/ci/playbooks/run-tests.yml@main] 2026-03-13 02:07:32.368038 | 2026-03-13 02:07:32.368152 | PLAY [Create hotstack scenario] 2026-03-13 02:07:32.388491 | 2026-03-13 02:07:32.388566 | TASK [Fetch bootstrap vars from remote to zuul executor] 2026-03-13 02:07:32.993418 | hotstack | ok 2026-03-13 02:07:33.004551 | 2026-03-13 02:07:33.004751 | TASK [Load bootstrap vars with template resolution] 2026-03-13 02:07:33.052368 | hotstack | ok 2026-03-13 02:07:33.065219 | 2026-03-13 02:07:33.065356 | TASK [Fetch test operator automation vars from remote to zuul executor] 2026-03-13 02:07:33.389491 | hotstack | changed 2026-03-13 02:07:33.399610 | 2026-03-13 02:07:33.399773 | TASK [Load test operator automation vars with template resolution] 2026-03-13 02:07:33.457300 | hotstack | ok 2026-03-13 02:07:33.469189 | 2026-03-13 02:07:33.469314 | TASK [Fetch stack outputs from remote to zuul executor] 2026-03-13 02:07:33.709567 | hotstack | Output suppressed because no_log was given 2026-03-13 02:07:33.721188 | 2026-03-13 02:07:33.721316 | TASK [Load stack output vars with template resolution] 2026-03-13 02:07:33.768569 | hotstack | Output suppressed because no_log was given 2026-03-13 02:07:33.781090 | 2026-03-13 02:07:33.781214 | TASK [Add Hotstack controller-0 to ansible inventory] 2026-03-13 02:07:33.818314 | hotstack | Output suppressed because no_log was given 2026-03-13 02:07:33.829090 | 2026-03-13 02:07:33.829218 | TASK [Wait for Hotstack controller-0 to be ready] 2026-03-13 02:07:39.429360 | hotstack -> 38.102.83.243 | Output suppressed because no_log was given 2026-03-13 02:07:39.440802 | 2026-03-13 02:07:39.440931 | TASK [Run scenario test automation] 2026-03-13 02:07:39.468746 | hotstack | ok 2026-03-13 02:07:39.513413 | 2026-03-13 02:07:39.513518 | TASK [hotloop : Initialize retry metrics] 2026-03-13 02:07:39.548377 | hotstack -> 38.102.83.243 | ok 2026-03-13 02:07:39.561262 | 2026-03-13 02:07:39.561390 | TASK [hotloop : Assert config is defined] 2026-03-13 02:07:39.658906 | hotstack -> 38.102.83.243 | ok: All assertions passed 2026-03-13 02:07:39.671057 | 2026-03-13 02:07:39.671185 | TASK [hotloop : Load stages] 2026-03-13 02:07:40.177091 | hotstack -> 38.102.83.243 | changed 2026-03-13 02:07:40.183947 | 2026-03-13 02:07:40.184013 | LOOP [hotloop : Ensure directory exists] 2026-03-13 02:07:40.664014 | hotstack -> 38.102.83.243 | ok: "/home/zuul/manifests" 2026-03-13 02:07:40.680535 | 2026-03-13 02:07:40.680764 | TASK [hotloop : Create temporary hotloop work directory] 2026-03-13 02:07:41.166482 | hotstack -> 38.102.83.243 | changed 2026-03-13 02:07:41.179951 | 2026-03-13 02:07:41.180092 | TASK [hotloop : Create temporary directory for templates] 2026-03-13 02:07:41.423531 | hotstack -> localhost | changed 2026-03-13 02:07:41.435956 | 2026-03-13 02:07:41.436089 | TASK [hotloop : Remove target directory for clean sync (local connection)] 2026-03-13 02:07:41.503518 | hotstack | skipping: Conditional result was False 2026-03-13 02:07:41.516408 | 2026-03-13 02:07:41.516538 | TASK [hotloop : Copy work files (local connection)] 2026-03-13 02:07:41.604336 | hotstack | skipping: Conditional result was False 2026-03-13 02:07:41.617720 | 2026-03-13 02:07:41.617857 | TASK [hotloop : Sync work files (remote connection) - using synchronize push] 2026-03-13 02:07:42.351026 | hotstack -> 38.102.83.155 | changed: 2026-03-13 02:07:42.351360 | hotstack -> 38.102.83.155 | .d..tp..... ./ 2026-03-13 02:07:42.351423 | hotstack -> 38.102.83.155 | 38.102.83.155 | localhost | ok: 2026-03-13 02:07:43.033226 | hotstack -> localhost | { 2026-03-13 02:07:43.033252 | hotstack -> localhost | "documentation": "Creates and configures a multiattach volume type in OpenStack that allows volumes to be attached to multiple instances simultaneously. This is essential for shared storage scenarios and high-availability configurations.", 2026-03-13 02:07:43.033271 | hotstack -> localhost | "name": "Set a multiattach volume type and create it if needed", 2026-03-13 02:07:43.033291 | hotstack -> localhost | "shell": "set -xe -o pipefail\noc project openstack\n\noc rsh openstackclient openstack volume type show multiattach &>/dev/null || \\\n oc rsh openstackclient openstack volume type create multiattach\n\noc rsh openstackclient openstack volume type set --property multiattach=\" True\" multiattach\n" 2026-03-13 02:07:43.033314 | hotstack -> localhost | } 2026-03-13 02:07:43.033344 | hotstack -> localhost | ok: All items complete 2026-03-13 02:07:43.033363 | 2026-03-13 02:07:43.498021 | hotstack -> localhost | ok: 2026-03-13 02:07:43.498159 | hotstack -> localhost | { 2026-03-13 02:07:43.498181 | hotstack -> localhost | "documentation": "Creates an external public network that provides connectivity to the outside world. This network uses a flat provider network type mapped to the 'datacentre' physical network and serves as the gateway for floating IP assignments.", 2026-03-13 02:07:43.498200 | hotstack -> localhost | "name": "Create public network if needed", 2026-03-13 02:07:43.498217 | hotstack -> localhost | "shell": "set -xe -o pipefail\noc project openstack\n\noc rsh openstackclient openstack network show public &>/dev/null || \\\n oc rsh openstackclient openstack network create public \\\n --external \\\n --no-share \\\n --default \\\n --provider-network-type flat \\\n --provider-physical-network datacentre\n" 2026-03-13 02:07:43.498237 | hotstack -> localhost | } 2026-03-13 02:07:43.987380 | hotstack -> localhost | ok: 2026-03-13 02:07:43.987496 | hotstack -> localhost | { 2026-03-13 02:07:43.987520 | hotstack -> localhost | "documentation": "Creates a subnet on the public network with a specific IP range (192.168.122.0/24) and allocation pool. This subnet provides DHCP services and defines the gateway for external connectivity. The allocation pool reserves IPs 171-250 for floating IPs.", 2026-03-13 02:07:43.987538 | hotstack -> localhost | "name": "Create subnet on public network if needed", 2026-03-13 02:07:43.987555 | hotstack -> localhost | "shell": "set -xe -o pipefail\noc project openstack\n\noc rsh openstackclient openstack subnet show public_subnet &>/dev/null || \\\n oc rsh openstackclient openstack subnet create public_subnet \\\n --network public \\\n --subnet-range 192.168.122.0/24 \\\n --allocation-pool start=192.168.122.171,end=192.168.122.250 \\\n --gateway 192.168.122.1 \\\n --dhcp\n" 2026-03-13 02:07:43.987580 | hotstack -> localhost | } 2026-03-13 02:07:44.506597 | hotstack -> localhost | ok: 2026-03-13 02:07:44.506822 | hotstack -> localhost | { 2026-03-13 02:07:44.506855 | hotstack -> localhost | "documentation": "Creates a shared private network for internal communication between OpenStack instances. This network is shared across tenants and provides isolated internal connectivity separate from the external public network.", 2026-03-13 02:07:44.506874 | hotstack -> localhost | "name": "Create private network if needed", 2026-03-13 02:07:44.506892 | hotstack -> localhost | "shell": "set -xe -o pipefail\noc project openstack\n\noc rsh openstackclient openstack network show private &>/dev/null || \\\n oc rsh openstackclient openstack network create private --share\n" 2026-03-13 02:07:44.506908 | hotstack -> localhost | } 2026-03-13 02:07:44.950720 | hotstack -> localhost | ok: 2026-03-13 02:07:44.950880 | hotstack -> localhost | { 2026-03-13 02:07:44.950930 | hotstack -> localhost | "documentation": "Creates a subnet on the private network with IP range 10.2.0.0/24 for internal instance communication. This subnet includes DHCP services and allocates IPs from 10.2.0.10 to 10.2.0.250 for instance assignments.", 2026-03-13 02:07:44.950970 | hotstack -> localhost | "name": "Create subnet on private network if needed", 2026-03-13 02:07:44.951014 | hotstack -> localhost | "shell": "set -xe -o pipefail\noc project openstack\n\noc rsh openstackclient openstack subnet show private_subnet &>/dev/null || \\\n oc rsh openstackclient openstack subnet create private_subnet \\\n --network private \\\n --subnet-range 10.2.0.0/24 \\\n --allocation-pool start=10.2.0.10,end=10.2.0.250 \\\n --gateway 10.2.0.1 \\\n --dhcp\n" 2026-03-13 02:07:44.951057 | hotstack -> localhost | } 2026-03-13 02:07:45.367733 | hotstack -> localhost | ok: 2026-03-13 02:07:45.367958 | hotstack -> localhost | { 2026-03-13 02:07:45.368021 | hotstack -> localhost | "documentation": "Executes comprehensive OpenStack validation tests using the Tempest framework.", 2026-03-13 02:07:45.368062 | hotstack -> localhost | "manifest": "tempest-tests.yml", 2026-03-13 02:07:45.368098 | hotstack -> localhost | "name": "Run tempest", 2026-03-13 02:07:45.368132 | hotstack -> localhost | "wait_conditions": [ 2026-03-13 02:07:45.368167 | hotstack -> localhost | "oc wait -n openstack tempests.test.openstack.org tempest-tests --for condition=ServiceConfigReady --timeout=120s" 2026-03-13 02:07:45.368201 | hotstack -> localhost | ], 2026-03-13 02:07:45.368234 | hotstack -> localhost | "wait_pod_completion": [ 2026-03-13 02:07:45.368267 | hotstack -> localhost | { 2026-03-13 02:07:45.368299 | hotstack -> localhost | "labels": { 2026-03-13 02:07:45.368331 | hotstack -> localhost | "operator": "test-operator", 2026-03-13 02:07:45.368362 | hotstack -> localhost | "service": "tempest", 2026-03-13 02:07:45.368395 | hotstack -> localhost | "workflowStep": "0" 2026-03-13 02:07:45.368427 | hotstack -> localhost | }, 2026-03-13 02:07:45.368459 | hotstack -> localhost | "namespace": "openstack", 2026-03-13 02:07:45.368491 | hotstack -> localhost | "poll_interval": 15, 2026-03-13 02:07:45.368523 | hotstack -> localhost | "timeout": 3600 2026-03-13 02:07:45.368555 | hotstack -> localhost | }, 2026-03-13 02:07:45.368587 | hotstack -> localhost | { 2026-03-13 02:07:45.368618 | hotstack -> localhost | "labels": { 2026-03-13 02:07:45.368651 | hotstack -> localhost | "operator": "test-operator", 2026-03-13 02:07:45.368748 | hotstack -> localhost | "service": "tempest", 2026-03-13 02:07:45.368786 | hotstack -> localhost | "workflowStep": "1" 2026-03-13 02:07:45.368818 | hotstack -> localhost | }, 2026-03-13 02:07:45.368860 | hotstack -> localhost | "namespace": "openstack", 2026-03-13 02:07:45.368894 | hotstack -> localhost | "poll_interval": 15, 2026-03-13 02:07:45.368925 | hotstack -> localhost | "timeout": 3600 2026-03-13 02:07:45.368958 | hotstack -> localhost | } 2026-03-13 02:07:45.368990 | hotstack -> localhost | ] 2026-03-13 02:07:45.369021 | hotstack -> localhost | } 2026-03-13 02:07:45.387503 | 2026-03-13 02:07:45.387639 | LOOP [hotloop : Execute automation stages] 2026-03-13 02:07:45.432271 | hotstack | ok: 2026-03-13 02:07:45.432793 | hotstack | { 2026-03-13 02:07:45.432873 | hotstack | "documentation": "Creates and configures a multiattach volume type in OpenStack that allows volumes to be attached to multiple instances simultaneously. This is essential for shared storage scenarios and high-availability configurations.", 2026-03-13 02:07:45.432921 | hotstack | "name": "Set a multiattach volume type and create it if needed", 2026-03-13 02:07:45.432960 | hotstack | "shell": "set -xe -o pipefail\noc project openstack\n\noc rsh openstackclient openstack volume type show multiattach &>/dev/null || \\\n oc rsh openstackclient openstack volume type create multiattach\n\noc rsh openstackclient openstack volume type set --property multiattach=\" True\" multiattach\n" 2026-03-13 02:07:45.432998 | hotstack | } 2026-03-13 02:07:45.444590 | hotstack | ok: 2026-03-13 02:07:45.444755 | hotstack | { 2026-03-13 02:07:45.444821 | hotstack | "documentation": "Creates an external public network that provides connectivity to the outside world. This network uses a flat provider network type mapped to the 'datacentre' physical network and serves as the gateway for floating IP assignments.", 2026-03-13 02:07:45.444872 | hotstack | "name": "Create public network if needed", 2026-03-13 02:07:45.444919 | hotstack | "shell": "set -xe -o pipefail\noc project openstack\n\noc rsh openstackclient openstack network show public &>/dev/null || \\\n oc rsh openstackclient openstack network create public \\\n --external \\\n --no-share \\\n --default \\\n --provider-network-type flat \\\n --provider-physical-network datacentre\n" 2026-03-13 02:07:45.444971 | hotstack | } 2026-03-13 02:07:45.456795 | hotstack | ok: 2026-03-13 02:07:45.456909 | hotstack | { 2026-03-13 02:07:45.456970 | hotstack | "documentation": "Creates a subnet on the public network with a specific IP range (192.168.122.0/24) and allocation pool. This subnet provides DHCP services and defines the gateway for external connectivity. The allocation pool reserves IPs 171-250 for floating IPs.", 2026-03-13 02:07:45.457023 | hotstack | "name": "Create subnet on public network if needed", 2026-03-13 02:07:45.457075 | hotstack | "shell": "set -xe -o pipefail\noc project openstack\n\noc rsh openstackclient openstack subnet show public_subnet &>/dev/null || \\\n oc rsh openstackclient openstack subnet create public_subnet \\\n --network public \\\n --subnet-range 192.168.122.0/24 \\\n --allocation-pool start=192.168.122.171,end=192.168.122.250 \\\n --gateway 192.168.122.1 \\\n --dhcp\n" 2026-03-13 02:07:45.457130 | hotstack | } 2026-03-13 02:07:45.468751 | hotstack | ok: 2026-03-13 02:07:45.468849 | hotstack | { 2026-03-13 02:07:45.468912 | hotstack | "documentation": "Creates a shared private network for internal communication between OpenStack instances. This network is shared across tenants and provides isolated internal connectivity separate from the external public network.", 2026-03-13 02:07:45.469008 | hotstack | "name": "Create private network if needed", 2026-03-13 02:07:45.469062 | hotstack | "shell": "set -xe -o pipefail\noc project openstack\n\noc rsh openstackclient openstack network show private &>/dev/null || \\\n oc rsh openstackclient openstack network create private --share\n" 2026-03-13 02:07:45.469106 | hotstack | } 2026-03-13 02:07:45.478248 | hotstack | ok: 2026-03-13 02:07:45.478338 | hotstack | { 2026-03-13 02:07:45.478402 | hotstack | "documentation": "Creates a subnet on the private network with IP range 10.2.0.0/24 for internal instance communication. This subnet includes DHCP services and allocates IPs from 10.2.0.10 to 10.2.0.250 for instance assignments.", 2026-03-13 02:07:45.478464 | hotstack | "name": "Create subnet on private network if needed", 2026-03-13 02:07:45.478495 | hotstack | "shell": "set -xe -o pipefail\noc project openstack\n\noc rsh openstackclient openstack subnet show private_subnet &>/dev/null || \\\n oc rsh openstackclient openstack subnet create private_subnet \\\n --network private \\\n --subnet-range 10.2.0.0/24 \\\n --allocation-pool start=10.2.0.10,end=10.2.0.250 \\\n --gateway 10.2.0.1 \\\n --dhcp\n" 2026-03-13 02:07:45.478546 | hotstack | } 2026-03-13 02:07:45.485624 | hotstack | ok: 2026-03-13 02:07:45.485721 | hotstack | { 2026-03-13 02:07:45.485752 | hotstack | "documentation": "Executes comprehensive OpenStack validation tests using the Tempest framework.", 2026-03-13 02:07:45.485778 | hotstack | "manifest": "tempest-tests.yml", 2026-03-13 02:07:45.485800 | hotstack | "name": "Run tempest", 2026-03-13 02:07:45.485822 | hotstack | "wait_conditions": [ 2026-03-13 02:07:45.485843 | hotstack | "oc wait -n openstack tempests.test.openstack.org tempest-tests --for condition=ServiceConfigReady --timeout=120s" 2026-03-13 02:07:45.485864 | hotstack | ], 2026-03-13 02:07:45.485885 | hotstack | "wait_pod_completion": [ 2026-03-13 02:07:45.485906 | hotstack | { 2026-03-13 02:07:45.485927 | hotstack | "labels": { 2026-03-13 02:07:45.485947 | hotstack | "operator": "test-operator", 2026-03-13 02:07:45.485967 | hotstack | "service": "tempest", 2026-03-13 02:07:45.485987 | hotstack | "workflowStep": "0" 2026-03-13 02:07:45.486007 | hotstack | }, 2026-03-13 02:07:45.486027 | hotstack | "namespace": "openstack", 2026-03-13 02:07:45.486047 | hotstack | "poll_interval": 15, 2026-03-13 02:07:45.486066 | hotstack | "timeout": 3600 2026-03-13 02:07:45.486086 | hotstack | }, 2026-03-13 02:07:45.486106 | hotstack | { 2026-03-13 02:07:45.486126 | hotstack | "labels": { 2026-03-13 02:07:45.486147 | hotstack | "operator": "test-operator", 2026-03-13 02:07:45.486168 | hotstack | "service": "tempest", 2026-03-13 02:07:45.486190 | hotstack | "workflowStep": "1" 2026-03-13 02:07:45.486213 | hotstack | }, 2026-03-13 02:07:45.486235 | hotstack | "namespace": "openstack", 2026-03-13 02:07:45.486256 | hotstack | "poll_interval": 15, 2026-03-13 02:07:45.486278 | hotstack | "timeout": 3600 2026-03-13 02:07:45.486299 | hotstack | } 2026-03-13 02:07:45.486321 | hotstack | ] 2026-03-13 02:07:45.486343 | hotstack | } 2026-03-13 02:07:45.511615 | hotstack | included: /var/lib/zuul/builds/46a5d1853f074c73b1d21511708ab398/work/tmp/ansible.8ktayw0khotloop_templates/stage_1.yml 2026-03-13 02:07:45.513830 | hotstack | included: /var/lib/zuul/builds/46a5d1853f074c73b1d21511708ab398/work/tmp/ansible.8ktayw0khotloop_templates/stage_2.yml 2026-03-13 02:07:45.515861 | hotstack | included: /var/lib/zuul/builds/46a5d1853f074c73b1d21511708ab398/work/tmp/ansible.8ktayw0khotloop_templates/stage_3.yml 2026-03-13 02:07:45.517893 | hotstack | included: /var/lib/zuul/builds/46a5d1853f074c73b1d21511708ab398/work/tmp/ansible.8ktayw0khotloop_templates/stage_4.yml 2026-03-13 02:07:45.519920 | hotstack | included: /var/lib/zuul/builds/46a5d1853f074c73b1d21511708ab398/work/tmp/ansible.8ktayw0khotloop_templates/stage_5.yml 2026-03-13 02:07:45.525266 | hotstack | included: /var/lib/zuul/builds/46a5d1853f074c73b1d21511708ab398/work/tmp/ansible.8ktayw0khotloop_templates/stage_6.yml 2026-03-13 02:07:45.535863 | 2026-03-13 02:07:45.535946 | TASK [hotloop : Stage: Set a multiattach volume type and create it if needed :: Run shell] 2026-03-12 22:07:45.994192 | controller-0 | + oc project openstack 2026-03-12 22:07:46.187423 | controller-0 | Now using project "openstack" on server "https://api.sno.openstack.lab:6443". 2026-03-12 22:07:46.190189 | controller-0 | + oc rsh openstackclient openstack volume type show multiattach 2026-03-12 22:07:49.438630 | controller-0 | + oc rsh openstackclient openstack volume type create multiattach 2026-03-12 22:07:51.444103 | controller-0 | +-------------+--------------------------------------+ 2026-03-12 22:07:51.652711 | controller-0 | | Field | Value | 2026-03-12 22:07:51.652816 | controller-0 | +-------------+--------------------------------------+ 2026-03-12 22:07:51.652832 | controller-0 | | description | None | 2026-03-12 22:07:51.652844 | controller-0 | | id | 8d412cc6-4205-49e7-bb8d-758fbefef9b7 | 2026-03-12 22:07:51.652855 | controller-0 | | is_public | True | 2026-03-12 22:07:51.652866 | controller-0 | | name | multiattach | 2026-03-12 22:07:51.652877 | controller-0 | +-------------+--------------------------------------+ 2026-03-12 22:07:51.652900 | controller-0 | + oc rsh openstackclient openstack volume type set --property 'multiattach= True' multiattach 2026-03-13 02:07:54.572111 | hotstack -> 38.102.83.243 | changed 2026-03-13 02:07:54.584935 | 2026-03-13 02:07:54.585066 | TASK [hotloop : Stage: Create public network if needed :: Run shell] 2026-03-12 22:07:55.111287 | controller-0 | + oc project openstack 2026-03-12 22:07:55.318374 | controller-0 | Already on project "openstack" on server "https://api.sno.openstack.lab:6443". 2026-03-12 22:07:55.321845 | controller-0 | + oc rsh openstackclient openstack network show public 2026-03-12 22:07:58.444717 | controller-0 | + oc rsh openstackclient openstack network create public --external --no-share --default --provider-network-type flat --provider-physical-network datacentre 2026-03-12 22:08:02.217914 | controller-0 | +---------------------------+--------------------------------------+ 2026-03-12 22:08:02.411825 | controller-0 | | Field | Value | 2026-03-12 22:08:02.411877 | controller-0 | +---------------------------+--------------------------------------+ 2026-03-12 22:08:02.411889 | controller-0 | | admin_state_up | UP | 2026-03-12 22:08:02.411894 | controller-0 | | availability_zone_hints | | 2026-03-12 22:08:02.411898 | controller-0 | | availability_zones | | 2026-03-12 22:08:02.411903 | controller-0 | | created_at | 2026-03-13T02:08:01Z | 2026-03-12 22:08:02.411907 | controller-0 | | description | | 2026-03-12 22:08:02.411912 | controller-0 | | dns_domain | | 2026-03-12 22:08:02.411916 | controller-0 | | id | 53b81dde-fe27-45a8-bdce-ed2910c91df1 | 2026-03-12 22:08:02.411921 | controller-0 | | ipv4_address_scope | None | 2026-03-12 22:08:02.411925 | controller-0 | | ipv6_address_scope | None | 2026-03-12 22:08:02.411930 | controller-0 | | is_default | True | 2026-03-12 22:08:02.411935 | controller-0 | | is_vlan_transparent | False | 2026-03-12 22:08:02.411940 | controller-0 | | l2_adjacency | True | 2026-03-12 22:08:02.411945 | controller-0 | | mtu | 1500 | 2026-03-12 22:08:02.411949 | controller-0 | | name | public | 2026-03-12 22:08:02.411953 | controller-0 | | port_security_enabled | True | 2026-03-12 22:08:02.411958 | controller-0 | | project_id | 17d510936237414ab1a97902e346cfd5 | 2026-03-12 22:08:02.411962 | controller-0 | | provider:network_type | flat | 2026-03-12 22:08:02.411967 | controller-0 | | provider:physical_network | datacentre | 2026-03-12 22:08:02.411971 | controller-0 | | provider:segmentation_id | None | 2026-03-12 22:08:02.411975 | controller-0 | | qos_policy_id | None | 2026-03-12 22:08:02.411980 | controller-0 | | revision_number | 1 | 2026-03-12 22:08:02.411984 | controller-0 | | router:external | External | 2026-03-12 22:08:02.411989 | controller-0 | | segments | None | 2026-03-12 22:08:02.411993 | controller-0 | | shared | False | 2026-03-12 22:08:02.411997 | controller-0 | | status | ACTIVE | 2026-03-12 22:08:02.412006 | controller-0 | | subnets | | 2026-03-12 22:08:02.412011 | controller-0 | | tags | | 2026-03-12 22:08:02.412016 | controller-0 | | tenant_id | 17d510936237414ab1a97902e346cfd5 | 2026-03-12 22:08:02.412021 | controller-0 | | updated_at | 2026-03-13T02:08:01Z | 2026-03-12 22:08:02.412026 | controller-0 | +---------------------------+--------------------------------------+ 2026-03-13 02:08:02.629647 | hotstack -> 38.102.83.243 | changed 2026-03-13 02:08:02.642707 | 2026-03-13 02:08:02.642846 | TASK [hotloop : Stage: Create subnet on public network if needed :: Run shell] 2026-03-12 22:08:03.165705 | controller-0 | + oc project openstack 2026-03-12 22:08:03.698697 | controller-0 | Already on project "openstack" on server "https://api.sno.openstack.lab:6443". 2026-03-12 22:08:03.702124 | controller-0 | + oc rsh openstackclient openstack subnet show public_subnet 2026-03-12 22:08:06.206592 | controller-0 | + oc rsh openstackclient openstack subnet create public_subnet --network public --subnet-range 192.168.122.0/24 --allocation-pool start=192.168.122.171,end=192.168.122.250 --gateway 192.168.122.1 --dhcp 2026-03-12 22:08:12.169250 | controller-0 | +----------------------+--------------------------------------+ 2026-03-12 22:08:12.362618 | controller-0 | | Field | Value | 2026-03-12 22:08:12.362683 | controller-0 | +----------------------+--------------------------------------+ 2026-03-12 22:08:12.362691 | controller-0 | | allocation_pools | 192.168.122.171-192.168.122.250 | 2026-03-12 22:08:12.362698 | controller-0 | | cidr | 192.168.122.0/24 | 2026-03-12 22:08:12.362704 | controller-0 | | created_at | 2026-03-13T02:08:10Z | 2026-03-12 22:08:12.362711 | controller-0 | | description | | 2026-03-12 22:08:12.362718 | controller-0 | | dns_nameservers | | 2026-03-12 22:08:12.362724 | controller-0 | | dns_publish_fixed_ip | None | 2026-03-12 22:08:12.362731 | controller-0 | | enable_dhcp | True | 2026-03-12 22:08:12.362737 | controller-0 | | gateway_ip | 192.168.122.1 | 2026-03-12 22:08:12.362743 | controller-0 | | host_routes | | 2026-03-12 22:08:12.362750 | controller-0 | | id | b578126b-f6a1-462b-b634-f368e50d87c7 | 2026-03-12 22:08:12.362756 | controller-0 | | ip_version | 4 | 2026-03-12 22:08:12.362762 | controller-0 | | ipv6_address_mode | None | 2026-03-12 22:08:12.362769 | controller-0 | | ipv6_ra_mode | None | 2026-03-12 22:08:12.362775 | controller-0 | | name | public_subnet | 2026-03-12 22:08:12.362782 | controller-0 | | network_id | 53b81dde-fe27-45a8-bdce-ed2910c91df1 | 2026-03-12 22:08:12.362788 | controller-0 | | project_id | 17d510936237414ab1a97902e346cfd5 | 2026-03-12 22:08:12.362795 | controller-0 | | revision_number | 0 | 2026-03-12 22:08:12.362801 | controller-0 | | segment_id | None | 2026-03-12 22:08:12.362808 | controller-0 | | service_types | | 2026-03-12 22:08:12.362814 | controller-0 | | subnetpool_id | None | 2026-03-12 22:08:12.362820 | controller-0 | | tags | | 2026-03-12 22:08:12.362827 | controller-0 | | updated_at | 2026-03-13T02:08:10Z | 2026-03-12 22:08:12.362833 | controller-0 | +----------------------+--------------------------------------+ 2026-03-13 02:08:12.687808 | hotstack -> 38.102.83.243 | changed 2026-03-13 02:08:12.696186 | 2026-03-13 02:08:12.696261 | TASK [hotloop : Stage: Create private network if needed :: Run shell] 2026-03-12 22:08:13.198865 | controller-0 | + oc project openstack 2026-03-12 22:08:13.406590 | controller-0 | Already on project "openstack" on server "https://api.sno.openstack.lab:6443". 2026-03-12 22:08:13.410325 | controller-0 | + oc rsh openstackclient openstack network show private 2026-03-12 22:08:16.246642 | controller-0 | + oc rsh openstackclient openstack network create private --share 2026-03-12 22:08:20.976057 | controller-0 | +---------------------------+--------------------------------------+ 2026-03-12 22:08:21.173717 | controller-0 | | Field | Value | 2026-03-12 22:08:21.173776 | controller-0 | +---------------------------+--------------------------------------+ 2026-03-12 22:08:21.173782 | controller-0 | | admin_state_up | UP | 2026-03-12 22:08:21.173787 | controller-0 | | availability_zone_hints | | 2026-03-12 22:08:21.173792 | controller-0 | | availability_zones | | 2026-03-12 22:08:21.173796 | controller-0 | | created_at | 2026-03-13T02:08:19Z | 2026-03-12 22:08:21.173801 | controller-0 | | description | | 2026-03-12 22:08:21.173806 | controller-0 | | dns_domain | | 2026-03-12 22:08:21.173810 | controller-0 | | id | eb7f2fc9-4d13-4045-b0d1-d0d6e35ff62f | 2026-03-12 22:08:21.173814 | controller-0 | | ipv4_address_scope | None | 2026-03-12 22:08:21.173819 | controller-0 | | ipv6_address_scope | None | 2026-03-12 22:08:21.173823 | controller-0 | | is_default | False | 2026-03-12 22:08:21.173828 | controller-0 | | is_vlan_transparent | False | 2026-03-12 22:08:21.173832 | controller-0 | | l2_adjacency | True | 2026-03-12 22:08:21.173837 | controller-0 | | mtu | 1442 | 2026-03-12 22:08:21.173841 | controller-0 | | name | private | 2026-03-12 22:08:21.173846 | controller-0 | | port_security_enabled | True | 2026-03-12 22:08:21.173850 | controller-0 | | project_id | 17d510936237414ab1a97902e346cfd5 | 2026-03-12 22:08:21.173855 | controller-0 | | provider:network_type | geneve | 2026-03-12 22:08:21.173859 | controller-0 | | provider:physical_network | None | 2026-03-12 22:08:21.173864 | controller-0 | | provider:segmentation_id | 53671 | 2026-03-12 22:08:21.173868 | controller-0 | | qos_policy_id | None | 2026-03-12 22:08:21.173872 | controller-0 | | revision_number | 1 | 2026-03-12 22:08:21.173877 | controller-0 | | router:external | Internal | 2026-03-12 22:08:21.173881 | controller-0 | | segments | None | 2026-03-12 22:08:21.173886 | controller-0 | | shared | True | 2026-03-12 22:08:21.173890 | controller-0 | | status | ACTIVE | 2026-03-12 22:08:21.173898 | controller-0 | | subnets | | 2026-03-12 22:08:21.173904 | controller-0 | | tags | | 2026-03-12 22:08:21.173908 | controller-0 | | tenant_id | 17d510936237414ab1a97902e346cfd5 | 2026-03-12 22:08:21.173913 | controller-0 | | updated_at | 2026-03-13T02:08:19Z | 2026-03-12 22:08:21.173918 | controller-0 | +---------------------------+--------------------------------------+ 2026-03-13 02:08:21.234734 | hotstack -> 38.102.83.243 | changed 2026-03-13 02:08:21.248121 | 2026-03-13 02:08:21.248262 | TASK [hotloop : Stage: Create subnet on private network if needed :: Run shell] 2026-03-12 22:08:21.778825 | controller-0 | + oc project openstack 2026-03-12 22:08:21.990387 | controller-0 | Already on project "openstack" on server "https://api.sno.openstack.lab:6443". 2026-03-12 22:08:21.994830 | controller-0 | + oc rsh openstackclient openstack subnet show private_subnet 2026-03-12 22:08:24.449596 | controller-0 | + oc rsh openstackclient openstack subnet create private_subnet --network private --subnet-range 10.2.0.0/24 --allocation-pool start=10.2.0.10,end=10.2.0.250 --gateway 10.2.0.1 --dhcp 2026-03-12 22:08:28.358041 | controller-0 | +----------------------+--------------------------------------+ 2026-03-12 22:08:28.509394 | controller-0 | | Field | Value | 2026-03-12 22:08:28.509450 | controller-0 | +----------------------+--------------------------------------+ 2026-03-12 22:08:28.509456 | controller-0 | | allocation_pools | 10.2.0.10-10.2.0.250 | 2026-03-12 22:08:28.509461 | controller-0 | | cidr | 10.2.0.0/24 | 2026-03-12 22:08:28.509466 | controller-0 | | created_at | 2026-03-13T02:08:27Z | 2026-03-12 22:08:28.509470 | controller-0 | | description | | 2026-03-12 22:08:28.509475 | controller-0 | | dns_nameservers | | 2026-03-12 22:08:28.509479 | controller-0 | | dns_publish_fixed_ip | None | 2026-03-12 22:08:28.509484 | controller-0 | | enable_dhcp | True | 2026-03-12 22:08:28.509488 | controller-0 | | gateway_ip | 10.2.0.1 | 2026-03-12 22:08:28.509492 | controller-0 | | host_routes | | 2026-03-12 22:08:28.509497 | controller-0 | | id | 17b8cf30-be9b-4f04-8564-bfd0ab451773 | 2026-03-12 22:08:28.509501 | controller-0 | | ip_version | 4 | 2026-03-12 22:08:28.509506 | controller-0 | | ipv6_address_mode | None | 2026-03-12 22:08:28.509510 | controller-0 | | ipv6_ra_mode | None | 2026-03-12 22:08:28.509515 | controller-0 | | name | private_subnet | 2026-03-12 22:08:28.509519 | controller-0 | | network_id | eb7f2fc9-4d13-4045-b0d1-d0d6e35ff62f | 2026-03-12 22:08:28.509524 | controller-0 | | project_id | 17d510936237414ab1a97902e346cfd5 | 2026-03-12 22:08:28.509528 | controller-0 | | revision_number | 0 | 2026-03-12 22:08:28.509532 | controller-0 | | segment_id | None | 2026-03-12 22:08:28.509537 | controller-0 | | service_types | | 2026-03-12 22:08:28.509558 | controller-0 | | subnetpool_id | None | 2026-03-12 22:08:28.509563 | controller-0 | | tags | | 2026-03-12 22:08:28.509567 | controller-0 | | updated_at | 2026-03-13T02:08:27Z | 2026-03-12 22:08:28.509572 | controller-0 | +----------------------+--------------------------------------+ 2026-03-13 02:08:28.790642 | hotstack -> 38.102.83.243 | changed 2026-03-13 02:08:28.803627 | 2026-03-13 02:08:28.803825 | TASK [hotloop : Stage: Run tempest :: Apply static manifest] 2026-03-13 02:08:28.832588 | hotstack | ok 2026-03-13 02:08:28.854850 | hotstack | included: /var/lib/zuul/builds/46a5d1853f074c73b1d21511708ab398/untrusted/project_0/github.com/openstack-k8s-operators/hotstack/roles/hotloop/tasks/static_manifest.yml 2026-03-13 02:08:28.867255 | 2026-03-13 02:08:28.867344 | TASK [hotloop : Stage: Run tempest :: Ensure manifest dir] 2026-03-13 02:08:29.401400 | hotstack -> 38.102.83.243 | ok 2026-03-13 02:08:29.415658 | 2026-03-13 02:08:29.415833 | TASK [hotloop : Stage: Run tempest :: Check if manifest exists in synced work directory] 2026-03-13 02:08:29.964031 | hotstack -> 38.102.83.243 | ok 2026-03-13 02:08:29.981333 | 2026-03-13 02:08:29.981511 | TASK [hotloop : Stage: Run tempest :: Copy manifest] 2026-03-13 02:08:30.660094 | hotstack -> 38.102.83.243 | changed 2026-03-13 02:08:30.670425 | 2026-03-13 02:08:30.670511 | LOOP [hotloop : Stage: Run tempest :: Apply patches] 2026-03-13 02:08:30.796732 | 2026-03-13 02:08:30.796875 | TASK [hotloop : Stage: Run tempest :: Apply static manifest] 2026-03-13 02:08:31.667942 | hotstack -> 38.102.83.243 | changed: Manifest file /home/zuul/manifests/tempest-tests.yml applied and saved as /home/zuul/manifests/tempest-tests.yml.applied with log at /home/zuul/manifests/tempest-tests.yml.applied.log 2026-03-13 02:08:31.680411 | 2026-03-13 02:08:31.680548 | LOOP [hotloop : Stage: Run tempest :: Wait conditions] 2026-03-13 02:08:32.716485 | hotstack -> 38.102.83.243 | ok: "oc wait -n openstack tempests.test.openstack.org tempest-tests --for condition=ServiceConfigReady --timeout=120s" 2026-03-13 02:08:32.716955 | 2026-03-13 02:08:32.730190 | 2026-03-13 02:08:32.730338 | LOOP [hotloop : Stage: Run tempest :: Wait for pod completion] 2026-03-13 02:10:30.005708 | hotstack -> 38.102.83.243 | ERROR: Item: {'labels': {'operator': 'test-operator', 'service': 'tempest', 'workflowStep': '0'}, 'namespace': 'openstack', 'poll_interval': 15, 'timeout': 3600} 2026-03-13 02:10:30.006302 | hotstack -> 38.102.83.243 | { 2026-03-13 02:10:30.006374 | hotstack -> 38.102.83.243 | "ansible_loop_var": "pod_wait", 2026-03-13 02:10:30.006417 | hotstack -> 38.102.83.243 | "attempts": 8, 2026-03-13 02:10:30.006455 | hotstack -> 38.102.83.243 | "elapsed_time": 106.44641375541687, 2026-03-13 02:10:30.006492 | hotstack -> 38.102.83.243 | "msg": "Failed to get pod status: Unable to connect to the server: net/http: TLS handshake timeout\n", 2026-03-13 02:10:30.006529 | hotstack -> 38.102.83.243 | "pod_name": "tempest-tests-s00-smoke-and-plugin-tests", 2026-03-13 02:10:30.006563 | hotstack -> 38.102.83.243 | "pod_wait": { 2026-03-13 02:10:30.006598 | hotstack -> 38.102.83.243 | "labels": { 2026-03-13 02:10:30.006631 | hotstack -> 38.102.83.243 | "operator": "test-operator", 2026-03-13 02:10:30.006700 | hotstack -> 38.102.83.243 | "service": "tempest", 2026-03-13 02:10:30.006748 | hotstack -> 38.102.83.243 | "workflowStep": "0" 2026-03-13 02:10:30.006784 | hotstack -> 38.102.83.243 | }, 2026-03-13 02:10:30.006818 | hotstack -> 38.102.83.243 | "namespace": "openstack", 2026-03-13 02:10:30.006851 | hotstack -> 38.102.83.243 | "poll_interval": 15, 2026-03-13 02:10:30.006885 | hotstack -> 38.102.83.243 | "timeout": 3600 2026-03-13 02:10:30.006919 | hotstack -> 38.102.83.243 | }, 2026-03-13 02:10:30.006964 | hotstack -> 38.102.83.243 | "status": "Running" 2026-03-13 02:10:30.006999 | hotstack -> 38.102.83.243 | } 2026-03-13 02:10:40.422302 | hotstack -> 38.102.83.243 | ERROR: Item: {'labels': {'operator': 'test-operator', 'service': 'tempest', 'workflowStep': '1'}, 'namespace': 'openstack', 'poll_interval': 15, 'timeout': 3600} 2026-03-13 02:10:40.422587 | hotstack -> 38.102.83.243 | { 2026-03-13 02:10:40.422655 | hotstack -> 38.102.83.243 | "ansible_loop_var": "pod_wait", 2026-03-13 02:10:40.422740 | hotstack -> 38.102.83.243 | "attempts": 1, 2026-03-13 02:10:40.422781 | hotstack -> 38.102.83.243 | "elapsed_time": 4.76837158203125e-07, 2026-03-13 02:10:40.422818 | hotstack -> 38.102.83.243 | "msg": "Failed to get pod status: Unable to connect to the server: net/http: TLS handshake timeout\n", 2026-03-13 02:10:40.422854 | hotstack -> 38.102.83.243 | "pod_name": "", 2026-03-13 02:10:40.422889 | hotstack -> 38.102.83.243 | "pod_wait": { 2026-03-13 02:10:40.422924 | hotstack -> 38.102.83.243 | "labels": { 2026-03-13 02:10:40.422958 | hotstack -> 38.102.83.243 | "operator": "test-operator", 2026-03-13 02:10:40.422992 | hotstack -> 38.102.83.243 | "service": "tempest", 2026-03-13 02:10:40.423026 | hotstack -> 38.102.83.243 | "workflowStep": "1" 2026-03-13 02:10:40.423061 | hotstack -> 38.102.83.243 | }, 2026-03-13 02:10:40.423095 | hotstack -> 38.102.83.243 | "namespace": "openstack", 2026-03-13 02:10:40.423128 | hotstack -> 38.102.83.243 | "poll_interval": 15, 2026-03-13 02:10:40.423161 | hotstack -> 38.102.83.243 | "timeout": 3600 2026-03-13 02:10:40.423195 | hotstack -> 38.102.83.243 | }, 2026-03-13 02:10:40.423230 | hotstack -> 38.102.83.243 | "status": "" 2026-03-13 02:10:40.423265 | hotstack -> 38.102.83.243 | } failure 2026-03-13 02:10:40.445625 | 2026-03-13 02:10:40.445782 | PLAY RECAP 2026-03-13 02:10:40.445869 | hotstack | ok: 33 changed: 13 unreachable: 0 failed: 1 skipped: 3 rescued: 0 ignored: 0 2026-03-13 02:10:40.445913 | 2026-03-13 02:10:40.572744 | RUN END RESULT_NORMAL: [untrusted : github.com/openstack-k8s-operators/hotstack/ci/playbooks/run-tests.yml@main] 2026-03-13 02:10:40.577478 | POST-RUN START: [trusted : review.rdoproject.org/config/playbooks/hotstack/post.yaml@master] 2026-03-13 02:10:41.162543 | 2026-03-13 02:10:41.162659 | PLAY [Collect hotstack logs] 2026-03-13 02:10:41.183479 | 2026-03-13 02:10:41.183549 | TASK [Fetch stack outputs from remote to zuul executor] 2026-03-13 02:10:41.738828 | hotstack | Output suppressed because no_log was given 2026-03-13 02:10:41.750324 | 2026-03-13 02:10:41.750483 | TASK [Load stack output vars with template resolution] 2026-03-13 02:10:41.800260 | hotstack | Output suppressed because no_log was given 2026-03-13 02:10:41.811748 | 2026-03-13 02:10:41.811884 | TASK [Add Hotstack controller-0 to ansible inventory] 2026-03-13 02:10:41.860408 | hotstack | Output suppressed because no_log was given 2026-03-13 02:10:41.871563 | 2026-03-13 02:10:41.871755 | TASK [Wait for Hotstack controller-0 to be ready] 2026-03-13 02:10:47.502518 | hotstack -> 38.102.83.243 | Output suppressed because no_log was given 2026-03-13 02:10:47.514085 | 2026-03-13 02:10:47.514216 | TASK [Collect hotstack logs] 2026-03-13 02:10:47.542492 | hotstack | Output suppressed because no_log was given 2026-03-13 02:10:47.593211 | 2026-03-13 02:10:47.593307 | TASK [hotlogs : Initialize failure tracking] 2026-03-13 02:10:47.624950 | hotstack | ok 2026-03-13 02:10:47.631731 | 2026-03-13 02:10:47.631819 | TASK [hotlogs : Ensure logs directory] 2026-03-13 02:10:48.034174 | hotstack | changed 2026-03-13 02:10:48.045644 | 2026-03-13 02:10:48.045839 | LOOP [hotlogs : Create hotlogs sub dirs] 2026-03-13 02:10:48.342068 | hotstack | changed: 2026-03-13 02:10:48.342765 | hotstack | { 2026-03-13 02:10:48.342852 | hotstack | "dest_dir": "ocp_cluster", 2026-03-13 02:10:48.342897 | hotstack | "src": "/home/zuul/ocp-cluster/.openshift_install.log" 2026-03-13 02:10:48.342935 | hotstack | } 2026-03-13 02:10:48.572334 | hotstack | ok: 2026-03-13 02:10:48.572486 | hotstack | { 2026-03-13 02:10:48.572542 | hotstack | "dest_dir": "ocp_cluster", 2026-03-13 02:10:48.572592 | hotstack | "src": "/home/zuul/cluster-custom-config/" 2026-03-13 02:10:48.572643 | hotstack | } 2026-03-13 02:10:48.825474 | hotstack | changed: 2026-03-13 02:10:48.825608 | hotstack | { 2026-03-13 02:10:48.825637 | hotstack | "dest_dir": "data", 2026-03-13 02:10:48.825680 | hotstack | "src": "/home/zuul/data/" 2026-03-13 02:10:48.825709 | hotstack | } 2026-03-13 02:10:49.015103 | hotstack | changed: 2026-03-13 02:10:49.015239 | hotstack | { 2026-03-13 02:10:49.015270 | hotstack | "dest_dir": "manifests", 2026-03-13 02:10:49.015296 | hotstack | "src": "/home/zuul/manifests/" 2026-03-13 02:10:49.015319 | hotstack | } 2026-03-13 02:10:49.201443 | hotstack | ok: 2026-03-13 02:10:49.201616 | hotstack | { 2026-03-13 02:10:49.201706 | hotstack | "dest_dir": ".", 2026-03-13 02:10:49.201763 | hotstack | "src": "/home/zuul/must-gather.tar.gz" 2026-03-13 02:10:49.201807 | hotstack | } 2026-03-13 02:10:49.436448 | hotstack | ok: 2026-03-13 02:10:49.436576 | hotstack | { 2026-03-13 02:10:49.436625 | hotstack | "dest_dir": ".", 2026-03-13 02:10:49.436701 | hotstack | "src": "/home/zuul/logs/tempest/" 2026-03-13 02:10:49.436748 | hotstack | } 2026-03-13 02:10:49.665155 | hotstack | changed: 2026-03-13 02:10:49.665352 | hotstack | { 2026-03-13 02:10:49.665420 | hotstack | "dest_dir": "nova-console-recordings", 2026-03-13 02:10:49.665462 | hotstack | "src": "/home/zuul/nova-console-recordings/" 2026-03-13 02:10:49.665500 | hotstack | } 2026-03-13 02:10:49.689327 | 2026-03-13 02:10:49.689524 | TASK [hotlogs : Run must-gather operations] 2026-03-13 02:11:00.270631 | hotstack -> 38.102.83.243 | ERROR 2026-03-13 02:11:00.270998 | hotstack -> 38.102.83.243 | { 2026-03-13 02:11:00.271061 | hotstack -> 38.102.83.243 | "msg": "Must-gather failed: error: unable to resolve image stream 'openshift/must-gather': Get \"https://api.sno.openstack.lab:6443/apis/image.openshift.io/v1/namespaces/openshift/imagestreams/must-gather\": net/http: TLS handshake timeout\n (Command: oc adm must-gather --image-stream=openshift/must-gather --image=quay.io/openstack-k8s-operators/openstack-must-gather --dest-dir=/home/zuul/must-gather --timeout=10m --host-network=false -- ADDITIONAL_NAMESPACES=sushy-emulator OPENSTACK_DATABASES=ALL SOS_EDPM=all SOS_DECOMPRESS=0 gather)", 2026-03-13 02:11:00.271112 | hotstack -> 38.102.83.243 | "rc": 1 2026-03-13 02:11:00.271153 | hotstack -> 38.102.83.243 | } 2026-03-13 02:11:00.283050 | 2026-03-13 02:11:00.283185 | TASK [hotlogs : Display must-gather failure] 2026-03-13 02:11:00.367573 | hotstack | ok: 2026-03-13 02:11:00.367829 | hotstack | Must-gather operations failed: Must-gather failed: error: unable to resolve image stream 'openshift/must-gather': Get "https://api.sno.openstack.lab:6443/apis/image.openshift.io/v1/namespaces/openshift/imagestreams/must-gather": net/http: TLS handshake timeout 2026-03-13 02:11:00.367909 | hotstack | (Command: oc adm must-gather --image-stream=openshift/must-gather --image=quay.io/openstack-k8s-operators/openstack-must-gather --dest-dir=/home/zuul/must-gather --timeout=10m --host-network=false -- ADDITIONAL_NAMESPACES=sushy-emulator OPENSTACK_DATABASES=ALL SOS_EDPM=all SOS_DECOMPRESS=0 gather) 2026-03-13 02:11:00.378210 | 2026-03-13 02:11:00.378338 | TASK [hotlogs : Track must-gather failure] 2026-03-13 02:11:00.444873 | hotstack | ok 2026-03-13 02:11:00.456377 | 2026-03-13 02:11:00.456507 | LOOP [hotlogs : Collect tempest logs from test-operator PVCs] 2026-03-13 02:11:11.061805 | hotstack -> 38.102.83.243 | ok: non-zero return code 2026-03-13 02:11:11.062239 | hotstack -> 38.102.83.243 | changed: All items complete 2026-03-13 02:11:11.062297 | 2026-03-13 02:11:11.077631 | 2026-03-13 02:11:11.077802 | LOOP [hotlogs : Display tempest log collection results] 2026-03-13 02:11:11.127785 | hotstack | ok: Namespace openstack: ['==========================================', 'Tempest Log Collection', '==========================================', 'Namespace: openstack', 'Logs directory: /home/zuul/logs/tempest', '==========================================', '', 'Searching for tempest log PVCs...', 'Unable to connect to the server: net/http: TLS handshake timeout'] 2026-03-13 02:11:11.140907 | 2026-03-13 02:11:11.141042 | LOOP [hotlogs : Track tempest log collection failures] 2026-03-13 02:11:11.173321 | hotstack | ok: 2026-03-13 02:11:11.173529 | hotstack | { 2026-03-13 02:11:11.173580 | hotstack | "ansible_loop_var": "item", 2026-03-13 02:11:11.173620 | hotstack | "changed": false, 2026-03-13 02:11:11.173657 | hotstack | "cmd": [ 2026-03-13 02:11:11.173731 | hotstack | "hotstack-collect-tempest-logs", 2026-03-13 02:11:11.173768 | hotstack | "--namespace", 2026-03-13 02:11:11.173803 | hotstack | "openstack" 2026-03-13 02:11:11.173838 | hotstack | ], 2026-03-13 02:11:11.173872 | hotstack | "delta": "0:00:10.144609", 2026-03-13 02:11:11.173905 | hotstack | "end": "2026-03-12 22:11:11.029705", 2026-03-13 02:11:11.173938 | hotstack | "failed": false, 2026-03-13 02:11:11.173970 | hotstack | "failed_when_result": false, 2026-03-13 02:11:11.174003 | hotstack | "invocation": { 2026-03-13 02:11:11.174036 | hotstack | "module_args": { 2026-03-13 02:11:11.174070 | hotstack | "_raw_params": "hotstack-collect-tempest-logs --namespace openstack", 2026-03-13 02:11:11.174103 | hotstack | "_uses_shell": false, 2026-03-13 02:11:11.174136 | hotstack | "argv": null, 2026-03-13 02:11:11.174169 | hotstack | "chdir": null, 2026-03-13 02:11:11.174201 | hotstack | "creates": null, 2026-03-13 02:11:11.174233 | hotstack | "executable": null, 2026-03-13 02:11:11.174265 | hotstack | "removes": null, 2026-03-13 02:11:11.174299 | hotstack | "stdin": null, 2026-03-13 02:11:11.174331 | hotstack | "stdin_add_newline": true, 2026-03-13 02:11:11.174364 | hotstack | "strip_empty_ends": true, 2026-03-13 02:11:11.174420 | hotstack | "warn": false, 2026-03-13 02:11:11.174453 | hotstack | "zuul_ansible_split_streams": false, 2026-03-13 02:11:11.174485 | hotstack | "zuul_log_id": "in-loop-ignore" 2026-03-13 02:11:11.174518 | hotstack | } 2026-03-13 02:11:11.174550 | hotstack | }, 2026-03-13 02:11:11.174583 | hotstack | "item": "openstack", 2026-03-13 02:11:11.174616 | hotstack | "msg": "non-zero return code", 2026-03-13 02:11:11.174649 | hotstack | "rc": 1, 2026-03-13 02:11:11.174735 | hotstack | "start": "2026-03-12 22:11:00.885096", 2026-03-13 02:11:11.174771 | hotstack | "stderr": "", 2026-03-13 02:11:11.174804 | hotstack | "stderr_lines": [], 2026-03-13 02:11:11.174837 | hotstack | "stdout": "==========================================\nTempest Log Collection\n==========================================\nNamespace: openstack\nLogs directory: /home/zuul/logs/tempest\n==========================================\n\nSearching for tempest log PVCs...\nUnable to connect to the server: net/http: TLS handshake timeout", 2026-03-13 02:11:11.174871 | hotstack | "stdout_lines": [ 2026-03-13 02:11:11.174904 | hotstack | "==========================================", 2026-03-13 02:11:11.174937 | hotstack | "Tempest Log Collection", 2026-03-13 02:11:11.174971 | hotstack | "==========================================", 2026-03-13 02:11:11.175004 | hotstack | "Namespace: openstack", 2026-03-13 02:11:11.175038 | hotstack | "Logs directory: /home/zuul/logs/tempest", 2026-03-13 02:11:11.175076 | hotstack | "==========================================", 2026-03-13 02:11:11.175111 | hotstack | "", 2026-03-13 02:11:11.175145 | hotstack | "Searching for tempest log PVCs...", 2026-03-13 02:11:11.175179 | hotstack | "Unable to connect to the server: net/http: TLS handshake timeout" 2026-03-13 02:11:11.175213 | hotstack | ], 2026-03-13 02:11:11.175247 | hotstack | "zuul_log_id": "in-loop-ignore" 2026-03-13 02:11:11.175280 | hotstack | } 2026-03-13 02:11:11.175337 | hotstack | ok: All items complete 2026-03-13 02:11:11.175379 | 2026-03-13 02:11:11.188033 | 2026-03-13 02:11:11.188167 | TASK [hotlogs : Create nova console recordings directory in base_dir on controller] 2026-03-13 02:11:11.560011 | hotstack -> 38.102.83.243 | changed 2026-03-13 02:11:11.572728 | 2026-03-13 02:11:11.572871 | TASK [hotlogs : Copy nova console recordings from NFS export] 2026-03-12 22:11:11.904097 | controller-0 | sending incremental file list 2026-03-12 22:11:11.905150 | controller-0 | ./ 2026-03-12 22:11:11.945837 | controller-0 | 2026-03-12 22:11:11.945886 | controller-0 | sent 66 bytes received 19 bytes 170.00 bytes/sec 2026-03-12 22:11:11.945892 | controller-0 | total size is 0 speedup is 0.00 2026-03-13 02:11:12.106229 | hotstack -> 38.102.83.243 | changed 2026-03-13 02:11:12.120192 | 2026-03-13 02:11:12.120326 | LOOP [hotlogs : Collect hotstack data] 2026-03-13 02:11:12.765937 | hotstack -> 38.102.83.155 | changed: >f+++++++++ .openshift_install.log 2026-03-13 02:11:13.271266 | hotstack -> 38.102.83.155 | changed: 2026-03-13 02:11:13.271420 | hotstack -> 38.102.83.155 | .d..t...... ./ 2026-03-13 02:11:13.271472 | hotstack -> 38.102.83.155 | cd+++++++++ butane/ 2026-03-13 02:11:13.271512 | hotstack -> 38.102.83.155 | >f+++++++++ butane/90-master-disable-netifnames.bu 2026-03-13 02:11:13.271551 | hotstack -> 38.102.83.155 | >f+++++++++ butane/90-master-enable-iscsi.bu 2026-03-13 02:11:13.271588 | hotstack -> 38.102.83.155 | >f+++++++++ butane/91-master-enable-multipath.bu 2026-03-13 02:11:13.271624 | hotstack -> 38.102.83.155 | >f+++++++++ butane/92-master-lv-cinder-volumes.bu 2026-03-13 02:11:13.271659 | hotstack -> 38.102.83.155 | cd+++++++++ config-assets/ 2026-03-13 02:11:13.271755 | hotstack -> 38.102.83.155 | >f+++++++++ config-assets/95-etcd_config.yaml 2026-03-13 02:11:13.271793 | hotstack -> 38.102.83.155 | >f+++++++++ config-assets/ovn_k8s_config.yaml 2026-03-13 02:11:13.271828 | hotstack -> 38.102.83.155 | cd+++++++++ machine-configs/ 2026-03-13 02:11:13.271863 | hotstack -> 38.102.83.155 | >f+++++++++ machine-configs/90-master-disable-netifnames.yaml 2026-03-13 02:11:13.271899 | hotstack -> 38.102.83.155 | >f+++++++++ machine-configs/90-master-enable-iscsi.yaml 2026-03-13 02:11:13.271934 | hotstack -> 38.102.83.155 | >f+++++++++ machine-configs/91-master-enable-multipath.yaml 2026-03-13 02:11:13.271969 | hotstack -> 38.102.83.155 | >f+++++++++ machine-configs/92-master-lv-cinder-volumes.yaml 2026-03-13 02:11:13.779593 | hotstack -> 38.102.83.155 | changed: .d..t...... ./ 2026-03-13 02:11:14.280505 | hotstack -> 38.102.83.155 | changed: 2026-03-13 02:11:14.280624 | hotstack -> 38.102.83.155 | .d..t...... ./ 2026-03-13 02:11:14.280712 | hotstack -> 38.102.83.155 | >f+++++++++ tempest-tests.yml.applied 2026-03-13 02:11:14.280758 | hotstack -> 38.102.83.155 | >f+++++++++ tempest-tests.yml.applied.log 2026-03-13 02:11:14.280798 | hotstack -> 38.102.83.155 | cd+++++++++ control-plane/ 2026-03-13 02:11:14.280836 | hotstack -> 38.102.83.155 | >f+++++++++ control-plane/control-plane.yaml 2026-03-13 02:11:14.280873 | hotstack -> 38.102.83.155 | >f+++++++++ control-plane/control-plane.yaml.applied 2026-03-13 02:11:14.280909 | hotstack -> 38.102.83.155 | >f+++++++++ control-plane/control-plane.yaml.applied.log 2026-03-13 02:11:14.280946 | hotstack -> 38.102.83.155 | >f+++++++++ control-plane/secrets.yaml.applied 2026-03-13 02:11:14.280984 | hotstack -> 38.102.83.155 | >f+++++++++ control-plane/secrets.yaml.applied.log 2026-03-13 02:11:14.281020 | hotstack -> 38.102.83.155 | cd+++++++++ dataplane/ 2026-03-13 02:11:14.281056 | hotstack -> 38.102.83.155 | >f+++++++++ dataplane/deployment-os.yaml.applied 2026-03-13 02:11:14.281092 | hotstack -> 38.102.83.155 | >f+++++++++ dataplane/deployment-os.yaml.applied.log 2026-03-13 02:11:14.281128 | hotstack -> 38.102.83.155 | >f+++++++++ dataplane/deployment-services.yaml.applied 2026-03-13 02:11:14.281164 | hotstack -> 38.102.83.155 | >f+++++++++ dataplane/deployment-services.yaml.applied.log 2026-03-13 02:11:14.281200 | hotstack -> 38.102.83.155 | >f+++++++++ dataplane/nodeset.yaml.applied 2026-03-13 02:11:14.281236 | hotstack -> 38.102.83.155 | >f+++++++++ dataplane/nodeset.yaml.applied.log 2026-03-13 02:11:14.281290 | hotstack -> 38.102.83.155 | >f+++++++++ dataplane/secrets.yaml.applied 2026-03-13 02:11:14.281327 | hotstack -> 38.102.83.155 | >f+++++++++ dataplane/secrets.yaml.applied.log 2026-03-13 02:11:14.281362 | hotstack -> 38.102.83.155 | cd+++++++++ deps/ 2026-03-13 02:11:14.281398 | hotstack -> 38.102.83.155 | >f+++++++++ deps/metallb.yaml.applied 2026-03-13 02:11:14.281433 | hotstack -> 38.102.83.155 | >f+++++++++ deps/metallb.yaml.applied.log 2026-03-13 02:11:14.281469 | hotstack -> 38.102.83.155 | >f+++++++++ deps/nmstate.yaml.applied 2026-03-13 02:11:14.281504 | hotstack -> 38.102.83.155 | >f+++++++++ deps/nmstate.yaml.applied.log 2026-03-13 02:11:14.281539 | hotstack -> 38.102.83.155 | >f+++++++++ deps/olm-deps.yaml.applied 2026-03-13 02:11:14.281573 | hotstack -> 38.102.83.155 | >f+++++++++ deps/olm-deps.yaml.applied.log 2026-03-13 02:11:14.281608 | hotstack -> 38.102.83.155 | >f+++++++++ deps/olm-topolvm.yaml.applied 2026-03-13 02:11:14.281644 | hotstack -> 38.102.83.155 | >f+++++++++ deps/olm-topolvm.yaml.applied.log 2026-03-13 02:11:14.281720 | hotstack -> 38.102.83.155 | cd+++++++++ networking/ 2026-03-13 02:11:14.281761 | hotstack -> 38.102.83.155 | >f+++++++++ networking/metallb.yaml.applied 2026-03-13 02:11:14.281798 | hotstack -> 38.102.83.155 | >f+++++++++ networking/metallb.yaml.applied.log 2026-03-13 02:11:14.281835 | hotstack -> 38.102.83.155 | >f+++++++++ networking/nad.yaml.applied 2026-03-13 02:11:14.281871 | hotstack -> 38.102.83.155 | >f+++++++++ networking/nad.yaml.applied.log 2026-03-13 02:11:14.281908 | hotstack -> 38.102.83.155 | >f+++++++++ networking/netconfig.yaml.applied 2026-03-13 02:11:14.281943 | hotstack -> 38.102.83.155 | >f+++++++++ networking/netconfig.yaml.applied.log 2026-03-13 02:11:14.281979 | hotstack -> 38.102.83.155 | >f+++++++++ networking/nncp.yaml.applied 2026-03-13 02:11:14.282014 | hotstack -> 38.102.83.155 | >f+++++++++ networking/nncp.yaml.applied.log 2026-03-13 02:11:14.282049 | hotstack -> 38.102.83.155 | cd+++++++++ olm/ 2026-03-13 02:11:14.282108 | hotstack -> 38.102.83.155 | >f+++++++++ olm/olm-openstack.yaml.applied 2026-03-13 02:11:14.282147 | hotstack -> 38.102.83.155 | >f+++++++++ olm/olm-openstack.yaml.applied.log 2026-03-13 02:11:14.282183 | hotstack -> 38.102.83.155 | >f+++++++++ olm/openstack-init.yaml.applied 2026-03-13 02:11:14.282219 | hotstack -> 38.102.83.155 | >f+++++++++ olm/openstack-init.yaml.applied.log 2026-03-13 02:11:14.282254 | hotstack -> 38.102.83.155 | cd+++++++++ topolvm/ 2026-03-13 02:11:14.282291 | hotstack -> 38.102.83.155 | >f+++++++++ topolvm/topolvmcluster.yaml.applied 2026-03-13 02:11:14.282328 | hotstack -> 38.102.83.155 | >f+++++++++ topolvm/topolvmcluster.yaml.applied.log 2026-03-13 02:11:14.865555 | hotstack -> 38.102.83.155 | ERROR: Item: {'src': '/home/zuul/must-gather.tar.gz', 'dest_dir': '.'} 2026-03-13 02:11:14.865761 | hotstack -> 38.102.83.155 | { 2026-03-13 02:11:14.865792 | hotstack -> 38.102.83.155 | "ansible_loop_var": "item", 2026-03-13 02:11:14.865811 | hotstack -> 38.102.83.155 | "item": { 2026-03-13 02:11:14.865828 | hotstack -> 38.102.83.155 | "dest_dir": ".", 2026-03-13 02:11:14.865843 | hotstack -> 38.102.83.155 | "src": "/home/zuul/must-gather.tar.gz" 2026-03-13 02:11:14.865859 | hotstack -> 38.102.83.155 | }, 2026-03-13 02:11:14.865877 | hotstack -> 38.102.83.155 | "msg": "Warning: Permanently added '38.102.83.243' (ED25519) to the list of known hosts.\r\nrsync: [sender] link_stat \"/home/zuul/must-gather.tar.gz\" failed: No such file or directory (2)\nrsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1841) [Receiver=3.2.5]\nrsync: [Receiver] write error: Broken pipe (32)\n", 2026-03-13 02:11:14.865899 | hotstack -> 38.102.83.155 | "rc": 23 2026-03-13 02:11:14.865914 | hotstack -> 38.102.83.155 | } 2026-03-13 02:11:15.364115 | hotstack -> 38.102.83.155 | changed: .d..t...... ./ 2026-03-13 02:11:15.856965 | hotstack -> 38.102.83.155 | changed: .d..t...... ./ 2026-03-13 02:11:15.879911 | hotstack | ERROR: Ignoring Errors 2026-03-13 02:11:15.896504 | 2026-03-13 02:11:15.896655 | LOOP [hotlogs : Check hotstack data collection failures] 2026-03-13 02:11:15.970783 | hotstack | skipping: Conditional result was False 2026-03-13 02:11:15.977606 | hotstack | skipping: Conditional result was False 2026-03-13 02:11:15.984069 | hotstack | skipping: Conditional result was False 2026-03-13 02:11:15.991331 | hotstack | skipping: Conditional result was False 2026-03-13 02:11:16.016822 | hotstack | ERROR 2026-03-13 02:11:16.017052 | hotstack | { 2026-03-13 02:11:16.017114 | hotstack | "msg": "Unexpected templating type error occurred on ({{ hotlogs_failures + ['Failed to collect: ' + item.item] }}): can only concatenate str (not \"dict\") to str. can only concatenate str (not \"dict\") to str" 2026-03-13 02:11:16.017158 | hotstack | } 2026-03-13 02:11:16.028062 | 2026-03-13 02:11:16.028197 | TASK [Print message that log collection failed] 2026-03-13 02:11:16.066579 | hotstack | ok: WARNING: Log collection failed and may be incomplete. Continue with cleanup ... 2026-03-13 02:11:16.102362 | 2026-03-13 02:11:16.102433 | PLAY RECAP 2026-03-13 02:11:16.102485 | hotstack | ok: 16 changed: 6 unreachable: 0 failed: 0 skipped: 0 rescued: 2 ignored: 1 2026-03-13 02:11:16.102512 | 2026-03-13 02:11:16.201835 | POST-RUN END RESULT_NORMAL: [trusted : review.rdoproject.org/config/playbooks/hotstack/post.yaml@master] 2026-03-13 02:11:16.204008 | POST-RUN START: [trusted : review.rdoproject.org/config/playbooks/hotstack/post-clean.yaml@master] 2026-03-13 02:11:16.740918 | 2026-03-13 02:11:16.741045 | PLAY [Clean up hotstack environment] 2026-03-13 02:11:16.763033 | 2026-03-13 02:11:16.763111 | TASK [Set _auth_type fact] 2026-03-13 02:11:16.812101 | hotstack | Output suppressed because no_log was given 2026-03-13 02:11:16.817370 | 2026-03-13 02:11:16.817441 | TASK [Write clouds.yaml] 2026-03-13 02:11:18.032015 | hotstack | Output suppressed because no_log was given 2026-03-13 02:11:18.043359 | 2026-03-13 02:11:18.043491 | TASK [Delete application credentials] 2026-03-13 02:11:21.593962 | hotstack | changed 2026-03-13 02:11:21.605829 | 2026-03-13 02:11:21.605970 | TASK [Print message that application credential failed to delete] 2026-03-13 02:11:21.633543 | hotstack | skipping: Conditional result was False 2026-03-13 02:11:21.645427 | 2026-03-13 02:11:21.645563 | TASK [Print message that application credential deleted successfully] 2026-03-13 02:11:21.693290 | hotstack | ok: Application credential deleted successfully. 2026-03-13 02:11:21.703553 | 2026-03-13 02:11:21.703744 | TASK [Write clouds.yaml with application credentials] 2026-03-13 02:11:21.741300 | 2026-03-13 02:11:21.741551 | TASK [Delete the heat stack] 2026-03-12 22:11:44.106026 | hotstack | 2026-03-13 02:11:23Z [hs-46a5d1853f074c73b1d21511708ab398]: DELETE_IN_PROGRESS Stack DELETE started 2026-03-12 22:11:44.106192 | hotstack | 2026-03-13 02:11:24Z [hs-46a5d1853f074c73b1d21511708ab398.compute0]: DELETE_IN_PROGRESS state changed 2026-03-12 22:11:44.106224 | hotstack | 2026-03-13 02:11:25Z [hs-46a5d1853f074c73b1d21511708ab398.ctlplane-net-router-interface]: DELETE_IN_PROGRESS state changed 2026-03-12 22:11:44.106242 | hotstack | 2026-03-13 02:11:26Z [hs-46a5d1853f074c73b1d21511708ab398.controller]: DELETE_IN_PROGRESS state changed 2026-03-12 22:11:44.106258 | hotstack | 2026-03-13 02:11:27Z [hs-46a5d1853f074c73b1d21511708ab398.master0]: DELETE_IN_PROGRESS state changed 2026-03-12 22:11:44.106274 | hotstack | 2026-03-13 02:11:28Z [hs-46a5d1853f074c73b1d21511708ab398.controller-floating-ip]: DELETE_IN_PROGRESS state changed 2026-03-12 22:11:44.106289 | hotstack | 2026-03-13 02:11:28Z [hs-46a5d1853f074c73b1d21511708ab398.ctlplane-net-router-interface]: DELETE_COMPLETE state changed 2026-03-12 22:11:44.106304 | hotstack | 2026-03-13 02:11:29Z [hs-46a5d1853f074c73b1d21511708ab398.controller-floating-ip]: DELETE_COMPLETE state changed 2026-03-12 22:11:44.106319 | hotstack | 2026-03-13 02:11:29Z [hs-46a5d1853f074c73b1d21511708ab398.machine-net-router-interface]: DELETE_IN_PROGRESS state changed 2026-03-12 22:11:44.106334 | hotstack | 2026-03-13 02:11:31Z [hs-46a5d1853f074c73b1d21511708ab398.compute0]: DELETE_COMPLETE state changed 2026-03-12 22:11:44.106432 | hotstack | 2026-03-13 02:11:31Z [hs-46a5d1853f074c73b1d21511708ab398.machine-net-router-interface]: DELETE_COMPLETE state changed 2026-03-12 22:11:44.106449 | hotstack | 2026-03-13 02:11:31Z [hs-46a5d1853f074c73b1d21511708ab398.compute0_init]: DELETE_IN_PROGRESS state changed 2026-03-12 22:11:44.106464 | hotstack | 2026-03-13 02:11:31Z [hs-46a5d1853f074c73b1d21511708ab398.compute0-trunk0]: DELETE_IN_PROGRESS state changed 2026-03-12 22:11:44.106479 | hotstack | 2026-03-13 02:11:31Z [hs-46a5d1853f074c73b1d21511708ab398.compute0_init]: DELETE_COMPLETE state changed 2026-03-12 22:11:44.106494 | hotstack | 2026-03-13 02:11:31Z [hs-46a5d1853f074c73b1d21511708ab398.router]: DELETE_IN_PROGRESS state changed 2026-03-12 22:11:44.106508 | hotstack | 2026-03-13 02:11:31Z [hs-46a5d1853f074c73b1d21511708ab398.compute0_hostname]: DELETE_IN_PROGRESS state changed 2026-03-12 22:11:44.106523 | hotstack | 2026-03-13 02:11:31Z [hs-46a5d1853f074c73b1d21511708ab398.dataplane_users]: DELETE_IN_PROGRESS state changed 2026-03-12 22:11:44.106538 | hotstack | 2026-03-13 02:11:32Z [hs-46a5d1853f074c73b1d21511708ab398.compute0_hostname]: DELETE_COMPLETE state changed 2026-03-12 22:11:44.106552 | hotstack | 2026-03-13 02:11:32Z [hs-46a5d1853f074c73b1d21511708ab398.dataplane_users]: DELETE_COMPLETE state changed 2026-03-12 22:11:44.106567 | hotstack | 2026-03-13 02:11:32Z [hs-46a5d1853f074c73b1d21511708ab398.compute0-trunk0]: DELETE_COMPLETE state changed 2026-03-12 22:11:44.106581 | hotstack | 2026-03-13 02:11:32Z [hs-46a5d1853f074c73b1d21511708ab398.compute0-storage-port]: DELETE_IN_PROGRESS state changed 2026-03-12 22:11:44.106596 | hotstack | 2026-03-13 02:11:32Z [hs-46a5d1853f074c73b1d21511708ab398.compute0-internal-api-port]: DELETE_IN_PROGRESS state changed 2026-03-12 22:11:44.106611 | hotstack | 2026-03-13 02:11:32Z [hs-46a5d1853f074c73b1d21511708ab398.compute0-ctlplane-trunk-parent-port]: DELETE_IN_PROGRESS state changed 2026-03-12 22:11:44.106625 | hotstack | 2026-03-13 02:11:32Z [hs-46a5d1853f074c73b1d21511708ab398.compute0-tenant-port]: DELETE_IN_PROGRESS state changed 2026-03-12 22:11:44.106640 | hotstack | 2026-03-13 02:11:33Z [hs-46a5d1853f074c73b1d21511708ab398.router]: DELETE_COMPLETE state changed 2026-03-12 22:11:44.106655 | hotstack | 2026-03-13 02:11:33Z [hs-46a5d1853f074c73b1d21511708ab398.compute0-internal-api-port]: DELETE_COMPLETE state changed 2026-03-12 22:11:44.106669 | hotstack | 2026-03-13 02:11:33Z [hs-46a5d1853f074c73b1d21511708ab398.compute0-ctlplane-trunk-parent-port]: DELETE_COMPLETE state changed 2026-03-12 22:11:44.106684 | hotstack | 2026-03-13 02:11:34Z [hs-46a5d1853f074c73b1d21511708ab398.compute0-tenant-port]: DELETE_COMPLETE state changed 2026-03-12 22:11:44.106699 | hotstack | 2026-03-13 02:11:35Z [hs-46a5d1853f074c73b1d21511708ab398.controller]: DELETE_COMPLETE state changed 2026-03-12 22:11:44.106713 | hotstack | 2026-03-13 02:11:35Z [hs-46a5d1853f074c73b1d21511708ab398.controller-init]: DELETE_IN_PROGRESS state changed 2026-03-12 22:11:44.106743 | hotstack | 2026-03-13 02:11:35Z [hs-46a5d1853f074c73b1d21511708ab398.controller-init]: DELETE_COMPLETE state changed 2026-03-12 22:11:44.106759 | hotstack | 2026-03-13 02:11:35Z [hs-46a5d1853f074c73b1d21511708ab398.controller-runcmd]: DELETE_IN_PROGRESS state changed 2026-03-12 22:11:44.106779 | hotstack | 2026-03-13 02:11:35Z [hs-46a5d1853f074c73b1d21511708ab398.controller_users]: DELETE_IN_PROGRESS state changed 2026-03-12 22:11:44.106793 | hotstack | 2026-03-13 02:11:35Z [hs-46a5d1853f074c73b1d21511708ab398.controller-write-files]: DELETE_IN_PROGRESS state changed 2026-03-12 22:11:44.106808 | hotstack | 2026-03-13 02:11:35Z [hs-46a5d1853f074c73b1d21511708ab398.controller-runcmd]: DELETE_COMPLETE state changed 2026-03-12 22:11:44.106822 | hotstack | 2026-03-13 02:11:35Z [hs-46a5d1853f074c73b1d21511708ab398.controller_users]: DELETE_COMPLETE state changed 2026-03-12 22:11:44.106851 | hotstack | 2026-03-13 02:11:35Z [hs-46a5d1853f074c73b1d21511708ab398.controller-write-files]: DELETE_COMPLETE state changed 2026-03-12 22:11:49.199398 | hotstack | 2026-03-13 02:11:35Z [hs-46a5d1853f074c73b1d21511708ab398.compute0-storage-port]: DELETE_COMPLETE state changed 2026-03-12 22:11:49.199505 | hotstack | 2026-03-13 02:11:39Z [hs-46a5d1853f074c73b1d21511708ab398.master0]: DELETE_COMPLETE state changed 2026-03-12 22:11:49.199520 | hotstack | 2026-03-13 02:11:39Z [hs-46a5d1853f074c73b1d21511708ab398.master0-machine-port]: DELETE_IN_PROGRESS state changed 2026-03-12 22:11:49.199531 | hotstack | 2026-03-13 02:11:39Z [hs-46a5d1853f074c73b1d21511708ab398.master0-cinder-vol1]: DELETE_IN_PROGRESS state changed 2026-03-12 22:11:49.199541 | hotstack | 2026-03-13 02:11:39Z [hs-46a5d1853f074c73b1d21511708ab398.master0-cinder-vol2]: DELETE_IN_PROGRESS state changed 2026-03-12 22:11:49.199551 | hotstack | 2026-03-13 02:11:39Z [hs-46a5d1853f074c73b1d21511708ab398.master0-lvms-vol0]: DELETE_IN_PROGRESS state changed 2026-03-12 22:11:49.199561 | hotstack | 2026-03-13 02:11:39Z [hs-46a5d1853f074c73b1d21511708ab398.master0-trunk0]: DELETE_IN_PROGRESS state changed 2026-03-12 22:11:49.199571 | hotstack | 2026-03-13 02:11:39Z [hs-46a5d1853f074c73b1d21511708ab398.master0-cinder-vol0]: DELETE_IN_PROGRESS state changed 2026-03-12 22:11:49.199588 | hotstack | 2026-03-13 02:11:40Z [hs-46a5d1853f074c73b1d21511708ab398.master0-lvms-vol0]: DELETE_COMPLETE state changed 2026-03-12 22:11:49.199600 | hotstack | 2026-03-13 02:11:41Z [hs-46a5d1853f074c73b1d21511708ab398.master0-machine-port]: DELETE_COMPLETE state changed 2026-03-12 22:11:49.199610 | hotstack | 2026-03-13 02:11:41Z [hs-46a5d1853f074c73b1d21511708ab398.extra-dhcp-opts-value]: DELETE_IN_PROGRESS state changed 2026-03-12 22:11:49.199620 | hotstack | 2026-03-13 02:11:41Z [hs-46a5d1853f074c73b1d21511708ab398.master0-trunk0]: DELETE_COMPLETE state changed 2026-03-12 22:11:49.199630 | hotstack | 2026-03-13 02:11:41Z [hs-46a5d1853f074c73b1d21511708ab398.extra-dhcp-opts-value]: DELETE_COMPLETE state changed 2026-03-12 22:11:49.199639 | hotstack | 2026-03-13 02:11:41Z [hs-46a5d1853f074c73b1d21511708ab398.master0-octavia-port]: DELETE_IN_PROGRESS state changed 2026-03-12 22:11:49.199649 | hotstack | 2026-03-13 02:11:41Z [hs-46a5d1853f074c73b1d21511708ab398.master0-tenant-port]: DELETE_IN_PROGRESS state changed 2026-03-12 22:11:49.199662 | hotstack | 2026-03-13 02:11:41Z [hs-46a5d1853f074c73b1d21511708ab398.controller-machine-port]: DELETE_IN_PROGRESS state changed 2026-03-12 22:11:49.199672 | hotstack | 2026-03-13 02:11:41Z [hs-46a5d1853f074c73b1d21511708ab398.master0-ctlplane-trunk-parent-port]: DELETE_IN_PROGRESS state changed 2026-03-12 22:11:49.199682 | hotstack | 2026-03-13 02:11:41Z [hs-46a5d1853f074c73b1d21511708ab398.master0-internal-api-port]: DELETE_IN_PROGRESS state changed 2026-03-12 22:11:49.199692 | hotstack | 2026-03-13 02:11:41Z [hs-46a5d1853f074c73b1d21511708ab398.master0-storage-port]: DELETE_IN_PROGRESS state changed 2026-03-12 22:11:49.199701 | hotstack | 2026-03-13 02:11:41Z [hs-46a5d1853f074c73b1d21511708ab398.master0-cinder-vol1]: DELETE_COMPLETE state changed 2026-03-12 22:11:49.199711 | hotstack | 2026-03-13 02:11:41Z [hs-46a5d1853f074c73b1d21511708ab398.master0-cinder-vol2]: DELETE_COMPLETE state changed 2026-03-12 22:11:49.199721 | hotstack | 2026-03-13 02:11:41Z [hs-46a5d1853f074c73b1d21511708ab398.master0-cinder-vol0]: DELETE_COMPLETE state changed 2026-03-12 22:11:49.199741 | hotstack | 2026-03-13 02:11:42Z [hs-46a5d1853f074c73b1d21511708ab398.master0-ctlplane-trunk-parent-port]: DELETE_COMPLETE state changed 2026-03-12 22:11:49.199751 | hotstack | 2026-03-13 02:11:42Z [hs-46a5d1853f074c73b1d21511708ab398.controller-machine-port]: DELETE_COMPLETE state changed 2026-03-12 22:11:49.199761 | hotstack | 2026-03-13 02:11:42Z [hs-46a5d1853f074c73b1d21511708ab398.ctlplane-subnet]: DELETE_IN_PROGRESS state changed 2026-03-12 22:11:49.199771 | hotstack | 2026-03-13 02:11:42Z [hs-46a5d1853f074c73b1d21511708ab398.master0-octavia-port]: DELETE_COMPLETE state changed 2026-03-12 22:11:49.199781 | hotstack | 2026-03-13 02:11:42Z [hs-46a5d1853f074c73b1d21511708ab398.machine-subnet]: DELETE_IN_PROGRESS state changed 2026-03-12 22:11:49.199797 | hotstack | 2026-03-13 02:11:42Z [hs-46a5d1853f074c73b1d21511708ab398.octavia-subnet]: DELETE_IN_PROGRESS state changed 2026-03-12 22:11:49.199807 | hotstack | 2026-03-13 02:11:42Z [hs-46a5d1853f074c73b1d21511708ab398.master0-storage-port]: DELETE_COMPLETE state changed 2026-03-12 22:11:49.199817 | hotstack | 2026-03-13 02:11:42Z [hs-46a5d1853f074c73b1d21511708ab398.storage-subnet]: DELETE_IN_PROGRESS state changed 2026-03-12 22:11:49.199827 | hotstack | 2026-03-13 02:11:42Z [hs-46a5d1853f074c73b1d21511708ab398.ctlplane-subnet]: DELETE_COMPLETE state changed 2026-03-12 22:11:49.199836 | hotstack | 2026-03-13 02:11:42Z [hs-46a5d1853f074c73b1d21511708ab398.ctlplane-net]: DELETE_IN_PROGRESS state changed 2026-03-12 22:11:49.199846 | hotstack | 2026-03-13 02:11:42Z [hs-46a5d1853f074c73b1d21511708ab398.master0-tenant-port]: DELETE_COMPLETE state changed 2026-03-12 22:11:49.199856 | hotstack | 2026-03-13 02:11:42Z [hs-46a5d1853f074c73b1d21511708ab398.octavia-subnet]: DELETE_COMPLETE state changed 2026-03-12 22:11:49.199865 | hotstack | 2026-03-13 02:11:42Z [hs-46a5d1853f074c73b1d21511708ab398.tenant-subnet]: DELETE_IN_PROGRESS state changed 2026-03-12 22:11:49.199875 | hotstack | 2026-03-13 02:11:43Z [hs-46a5d1853f074c73b1d21511708ab398.octavia-net]: DELETE_IN_PROGRESS state changed 2026-03-12 22:11:49.199885 | hotstack | 2026-03-13 02:11:43Z [hs-46a5d1853f074c73b1d21511708ab398.master0-internal-api-port]: DELETE_COMPLETE state changed 2026-03-12 22:11:49.199901 | hotstack | 2026-03-13 02:11:43Z [hs-46a5d1853f074c73b1d21511708ab398.internal-api-subnet]: DELETE_IN_PROGRESS state changed 2026-03-12 22:11:49.278084 | hotstack | 2026-03-13 02:11:43Z [hs-46a5d1853f074c73b1d21511708ab398.tenant-subnet]: DELETE_COMPLETE state changed 2026-03-12 22:11:49.278276 | hotstack | 2026-03-13 02:11:43Z [hs-46a5d1853f074c73b1d21511708ab398.ctlplane-net]: DELETE_COMPLETE state changed 2026-03-12 22:11:49.278294 | hotstack | 2026-03-13 02:11:43Z [hs-46a5d1853f074c73b1d21511708ab398.tenant-net]: DELETE_IN_PROGRESS state changed 2026-03-12 22:11:49.278304 | hotstack | 2026-03-13 02:11:44Z [hs-46a5d1853f074c73b1d21511708ab398.internal-api-subnet]: DELETE_COMPLETE state changed 2026-03-12 22:11:49.278314 | hotstack | 2026-03-13 02:11:44Z [hs-46a5d1853f074c73b1d21511708ab398.octavia-net]: DELETE_COMPLETE state changed 2026-03-12 22:11:49.278324 | hotstack | 2026-03-13 02:11:44Z [hs-46a5d1853f074c73b1d21511708ab398.internal-api-net]: DELETE_IN_PROGRESS state changed 2026-03-12 22:11:49.278333 | hotstack | 2026-03-13 02:11:44Z [hs-46a5d1853f074c73b1d21511708ab398.machine-subnet]: DELETE_COMPLETE state changed 2026-03-12 22:11:49.278342 | hotstack | 2026-03-13 02:11:44Z [hs-46a5d1853f074c73b1d21511708ab398.machine-net]: DELETE_IN_PROGRESS state changed 2026-03-12 22:11:49.278352 | hotstack | 2026-03-13 02:11:44Z [hs-46a5d1853f074c73b1d21511708ab398.tenant-net]: DELETE_COMPLETE state changed 2026-03-12 22:11:49.278361 | hotstack | 2026-03-13 02:11:45Z [hs-46a5d1853f074c73b1d21511708ab398.internal-api-net]: DELETE_COMPLETE state changed 2026-03-12 22:11:49.278371 | hotstack | 2026-03-13 02:11:45Z [hs-46a5d1853f074c73b1d21511708ab398.machine-net]: DELETE_COMPLETE state changed 2026-03-12 22:11:49.278381 | hotstack | 2026-03-13 02:11:47Z [hs-46a5d1853f074c73b1d21511708ab398.storage-subnet]: DELETE_COMPLETE state changed 2026-03-12 22:11:49.278390 | hotstack | 2026-03-13 02:11:47Z [hs-46a5d1853f074c73b1d21511708ab398.storage-net]: DELETE_IN_PROGRESS state changed 2026-03-12 22:11:49.278413 | hotstack | 2026-03-13 02:11:48Z [hs-46a5d1853f074c73b1d21511708ab398.storage-net]: DELETE_COMPLETE state changed 2026-03-12 22:11:49.278431 | hotstack | 2026-03-13 02:11:48Z [hs-46a5d1853f074c73b1d21511708ab398]: DELETE_COMPLETE Stack DELETE completed successfully 2026-03-13 02:11:49.322054 | hotstack | changed 2026-03-13 02:11:49.334076 | 2026-03-13 02:11:49.334201 | TASK [Print message that stack failed to delete] 2026-03-13 02:11:49.361363 | hotstack | skipping: Conditional result was False 2026-03-13 02:11:49.376237 | 2026-03-13 02:11:49.376367 | TASK [Print message that stack deleted successfully] 2026-03-13 02:11:49.423830 | hotstack | ok: Stack deleted successfully. 2026-03-13 02:11:49.434308 | 2026-03-13 02:11:49.434435 | TASK [Delete clouds.yaml] 2026-03-13 02:11:49.840051 | hotstack | Output suppressed because no_log was given 2026-03-13 02:11:49.881482 | 2026-03-13 02:11:49.881553 | PLAY RECAP 2026-03-13 02:11:49.881606 | hotstack | ok: 7 changed: 4 unreachable: 0 failed: 0 skipped: 3 rescued: 0 ignored: 0 2026-03-13 02:11:49.881634 | 2026-03-13 02:11:49.976088 | POST-RUN END RESULT_NORMAL: [trusted : review.rdoproject.org/config/playbooks/hotstack/post-clean.yaml@master] 2026-03-13 02:11:49.978319 | POST-RUN START: [trusted : review.rdoproject.org/config/playbooks/base-minimal/post-ssh.yaml@master] 2026-03-13 02:11:50.526579 | 2026-03-13 02:11:50.526710 | PLAY [all] 2026-03-13 02:11:50.545555 | 2026-03-13 02:11:50.545626 | TASK [include_role : fetch-output] 2026-03-13 02:11:50.584557 | hotstack | ok 2026-03-13 02:11:50.600772 | 2026-03-13 02:11:50.600845 | TASK [fetch-output : Set log path for multiple nodes] 2026-03-13 02:11:50.665708 | hotstack | skipping: Conditional result was False 2026-03-13 02:11:50.677747 | 2026-03-13 02:11:50.677884 | TASK [fetch-output : Set log path for single node] 2026-03-13 02:11:50.724371 | hotstack | ok 2026-03-13 02:11:50.733883 | 2026-03-13 02:11:50.733995 | LOOP [fetch-output : Ensure local output dirs] 2026-03-13 02:11:51.142603 | hotstack -> localhost | ok: "/var/lib/zuul/builds/46a5d1853f074c73b1d21511708ab398/work/logs" 2026-03-13 02:11:51.407938 | hotstack -> localhost | changed: "/var/lib/zuul/builds/46a5d1853f074c73b1d21511708ab398/work/artifacts" 2026-03-13 02:11:51.709445 | hotstack -> localhost | changed: "/var/lib/zuul/builds/46a5d1853f074c73b1d21511708ab398/work/docs" 2026-03-13 02:11:51.731508 | 2026-03-13 02:11:51.731721 | LOOP [fetch-output : Collect logs, artifacts and docs] 2026-03-13 02:11:52.463427 | hotstack | changed: 2026-03-13 02:11:52.463812 | hotstack | .d..t...... ./ 2026-03-13 02:11:52.463892 | hotstack | cd+++++++++ hotlogs/ 2026-03-13 02:11:52.463938 | hotstack | cd+++++++++ hotlogs/data/ 2026-03-13 02:11:52.463979 | hotstack | cd+++++++++ hotlogs/manifests/ 2026-03-13 02:11:52.464023 | hotstack | >f+++++++++ hotlogs/manifests/tempest-tests.yml.applied 2026-03-13 02:11:52.464064 | hotstack | >f+++++++++ hotlogs/manifests/tempest-tests.yml.applied.log 2026-03-13 02:11:52.464102 | hotstack | cd+++++++++ hotlogs/manifests/control-plane/ 2026-03-13 02:11:52.464140 | hotstack | >f+++++++++ hotlogs/manifests/control-plane/control-plane.yaml 2026-03-13 02:11:52.464178 | hotstack | >f+++++++++ hotlogs/manifests/control-plane/control-plane.yaml.applied 2026-03-13 02:11:52.464214 | hotstack | >f+++++++++ hotlogs/manifests/control-plane/control-plane.yaml.applied.log 2026-03-13 02:11:52.464251 | hotstack | >f+++++++++ hotlogs/manifests/control-plane/secrets.yaml.applied 2026-03-13 02:11:52.464287 | hotstack | >f+++++++++ hotlogs/manifests/control-plane/secrets.yaml.applied.log 2026-03-13 02:11:52.464322 | hotstack | cd+++++++++ hotlogs/manifests/dataplane/ 2026-03-13 02:11:52.464358 | hotstack | >f+++++++++ hotlogs/manifests/dataplane/deployment-os.yaml.applied 2026-03-13 02:11:52.464393 | hotstack | >f+++++++++ hotlogs/manifests/dataplane/deployment-os.yaml.applied.log 2026-03-13 02:11:52.464434 | hotstack | >f+++++++++ hotlogs/manifests/dataplane/deployment-services.yaml.applied 2026-03-13 02:11:52.464470 | hotstack | >f+++++++++ hotlogs/manifests/dataplane/deployment-services.yaml.applied.log 2026-03-13 02:11:52.464506 | hotstack | >f+++++++++ hotlogs/manifests/dataplane/nodeset.yaml.applied 2026-03-13 02:11:52.464541 | hotstack | >f+++++++++ hotlogs/manifests/dataplane/nodeset.yaml.applied.log 2026-03-13 02:11:52.464586 | hotstack | >f+++++++++ hotlogs/manifests/dataplane/secrets.yaml.applied 2026-03-13 02:11:52.464621 | hotstack | >f+++++++++ hotlogs/manifests/dataplane/secrets.yaml.applied.log 2026-03-13 02:11:52.464655 | hotstack | cd+++++++++ hotlogs/manifests/deps/ 2026-03-13 02:11:52.464731 | hotstack | >f+++++++++ hotlogs/manifests/deps/metallb.yaml.applied 2026-03-13 02:11:52.464769 | hotstack | >f+++++++++ hotlogs/manifests/deps/metallb.yaml.applied.log 2026-03-13 02:11:52.464806 | hotstack | >f+++++++++ hotlogs/manifests/deps/nmstate.yaml.applied 2026-03-13 02:11:52.464842 | hotstack | >f+++++++++ hotlogs/manifests/deps/nmstate.yaml.applied.log 2026-03-13 02:11:52.464898 | hotstack | >f+++++++++ hotlogs/manifests/deps/olm-deps.yaml.applied 2026-03-13 02:11:52.464934 | hotstack | >f+++++++++ hotlogs/manifests/deps/olm-deps.yaml.applied.log 2026-03-13 02:11:52.464968 | hotstack | >f+++++++++ hotlogs/manifests/deps/olm-topolvm.yaml.applied 2026-03-13 02:11:52.465003 | hotstack | >f+++++++++ hotlogs/manifests/deps/olm-topolvm.yaml.applied.log 2026-03-13 02:11:52.465038 | hotstack | cd+++++++++ hotlogs/manifests/networking/ 2026-03-13 02:11:52.465073 | hotstack | >f+++++++++ hotlogs/manifests/networking/metallb.yaml.applied 2026-03-13 02:11:52.465108 | hotstack | >f+++++++++ hotlogs/manifests/networking/metallb.yaml.applied.log 2026-03-13 02:11:52.465142 | hotstack | >f+++++++++ hotlogs/manifests/networking/nad.yaml.applied 2026-03-13 02:11:52.465176 | hotstack | >f+++++++++ hotlogs/manifests/networking/nad.yaml.applied.log 2026-03-13 02:11:52.465210 | hotstack | >f+++++++++ hotlogs/manifests/networking/netconfig.yaml.applied 2026-03-13 02:11:52.465243 | hotstack | >f+++++++++ hotlogs/manifests/networking/netconfig.yaml.applied.log 2026-03-13 02:11:52.465276 | hotstack | >f+++++++++ hotlogs/manifests/networking/nncp.yaml.applied 2026-03-13 02:11:52.465313 | hotstack | >f+++++++++ hotlogs/manifests/networking/nncp.yaml.applied.log 2026-03-13 02:11:52.465350 | hotstack | cd+++++++++ hotlogs/manifests/olm/ 2026-03-13 02:11:52.465385 | hotstack | >f+++++++++ hotlogs/manifests/olm/olm-openstack.yaml.applied 2026-03-13 02:11:52.465420 | hotstack | >f+++++++++ hotlogs/manifests/olm/olm-openstack.yaml.applied.log 2026-03-13 02:11:52.465455 | hotstack | >f+++++++++ hotlogs/manifests/olm/openstack-init.yaml.applied 2026-03-13 02:11:52.465490 | hotstack | >f+++++++++ hotlogs/manifests/olm/openstack-init.yaml.applied.log 2026-03-13 02:11:52.465524 | hotstack | cd+++++++++ hotlogs/manifests/topolvm/ 2026-03-13 02:11:52.465559 | hotstack | >f+++++++++ hotlogs/manifests/topolvm/topolvmcluster.yaml.applied 2026-03-13 02:11:52.465597 | hotstack | >f+++++++++ hotlogs/manifests/topolvm/topolvmcluster.yaml.applied.log 2026-03-13 02:11:52.465635 | hotstack | cd+++++++++ hotlogs/nova-console-recordings/ 2026-03-13 02:11:52.465735 | hotstack | cd+++++++++ hotlogs/ocp_cluster/ 2026-03-13 02:11:52.465779 | hotstack | >f+++++++++ hotlogs/ocp_cluster/openshift_install.log 2026-03-13 02:11:52.465816 | hotstack | cd+++++++++ hotlogs/ocp_cluster/butane/ 2026-03-13 02:11:52.465851 | hotstack | >f+++++++++ hotlogs/ocp_cluster/butane/90-master-disable-netifnames.bu 2026-03-13 02:11:52.465886 | hotstack | >f+++++++++ hotlogs/ocp_cluster/butane/90-master-enable-iscsi.bu 2026-03-13 02:11:52.465921 | hotstack | >f+++++++++ hotlogs/ocp_cluster/butane/91-master-enable-multipath.bu 2026-03-13 02:11:52.465957 | hotstack | >f+++++++++ hotlogs/ocp_cluster/butane/92-master-lv-cinder-volumes.bu 2026-03-13 02:11:52.465992 | hotstack | cd+++++++++ hotlogs/ocp_cluster/config-assets/ 2026-03-13 02:11:52.466027 | hotstack | >f+++++++++ hotlogs/ocp_cluster/config-assets/95-etcd_config.yaml 2026-03-13 02:11:52.466062 | hotstack | >f+++++++++ hotlogs/ocp_cluster/config-assets/ovn_k8s_config.yaml 2026-03-13 02:11:52.466096 | hotstack | cd+++++++++ hotlogs/ocp_cluster/machine-configs/ 2026-03-13 02:11:52.466132 | hotstack | >f+++++++++ hotlogs/ocp_cluster/machine-configs/90-master-disable-netifnames.yaml 2026-03-13 02:11:52.466167 | hotstack | >f+++++++++ hotlogs/ocp_cluster/machine-configs/90-master-enable-iscsi.yaml 2026-03-13 02:11:52.466212 | hotstack | >f+++++++++ hotlogs/ocp_cluster/machine-configs/91-master-enable-multipath.yaml 2026-03-13 02:11:52.466248 | hotstack | >f+++++++++ hotlogs/ocp_cluster/machine-configs/92-master-lv-cinder-volumes.yaml 2026-03-13 02:11:52.466320 | hotstack | changed: All items complete 2026-03-13 02:11:52.466364 | 2026-03-13 02:11:52.983042 | hotstack | changed: .d..t...... ./ 2026-03-13 02:11:53.509711 | hotstack | changed: .d..t...... ./ 2026-03-13 02:11:53.536275 | 2026-03-13 02:11:53.536470 | TASK [include_role : fetch-output-openshift] 2026-03-13 02:11:53.563489 | hotstack | skipping: Conditional result was False 2026-03-13 02:11:53.576142 | 2026-03-13 02:11:53.576282 | LOOP [merge-output-to-logs : Move artifacts and docs to logs dir] 2026-03-13 02:11:54.049134 | hotstack -> localhost | ok: Item: artifacts Runtime: 0:00:00.012648 2026-03-13 02:11:54.327404 | hotstack -> localhost | ok: Item: docs Runtime: 0:00:00.012445 2026-03-13 02:11:54.385209 | 2026-03-13 02:11:54.385330 | PLAY [all] 2026-03-13 02:11:54.404408 | 2026-03-13 02:11:54.404496 | TASK [remove-build-sshkey : Remove the build SSH key from all nodes] 2026-03-13 02:11:55.050101 | hotstack | changed 2026-03-13 02:11:55.097527 | 2026-03-13 02:11:55.097603 | PLAY RECAP 2026-03-13 02:11:55.097657 | hotstack | ok: 5 changed: 4 unreachable: 0 failed: 0 skipped: 2 rescued: 0 ignored: 0 2026-03-13 02:11:55.097715 | 2026-03-13 02:11:55.197508 | POST-RUN END RESULT_NORMAL: [trusted : review.rdoproject.org/config/playbooks/base-minimal/post-ssh.yaml@master] 2026-03-13 02:11:55.199653 | POST-RUN START: [trusted : review.rdoproject.org/config/playbooks/base-minimal/post-logs.yaml@master] 2026-03-13 02:11:55.794815 | 2026-03-13 02:11:55.794990 | PLAY [localhost] 2026-03-13 02:11:55.811807 | 2026-03-13 02:11:55.811881 | TASK [generate-zuul-manifest : Generate Zuul manifest] 2026-03-13 02:11:56.234991 | localhost | changed 2026-03-13 02:11:56.245156 | 2026-03-13 02:11:56.245319 | TASK [generate-zuul-manifest : Return Zuul manifest URL to Zuul] 2026-03-13 02:11:56.281505 | localhost | ok 2026-03-13 02:11:56.299378 | 2026-03-13 02:11:56.299533 | TASK [add-fileserver : Create SSH private key tempfile] 2026-03-13 02:11:56.690788 | localhost | changed 2026-03-13 02:11:56.700968 | 2026-03-13 02:11:56.701112 | TASK [add-fileserver : Create SSH private key from secret] 2026-03-13 02:11:57.408785 | localhost | changed 2026-03-13 02:11:57.419370 | 2026-03-13 02:11:57.419507 | TASK [add-fileserver : Add fileserver ssh key] 2026-03-13 02:11:57.872208 | localhost | Identity added: /var/lib/zuul/builds/46a5d1853f074c73b1d21511708ab398/work/tmp/ansible.x_t5p_7n (/var/lib/zuul/builds/46a5d1853f074c73b1d21511708ab398/work/tmp/ansible.x_t5p_7n) 2026-03-13 02:11:57.872505 | localhost | ok: Runtime: 0:00:00.012509 2026-03-13 02:11:57.881858 | 2026-03-13 02:11:57.881994 | TASK [add-fileserver : Remove SSH private key from disk] 2026-03-13 02:11:58.191781 | localhost | ok: Runtime: 0:00:00.008055 2026-03-13 02:11:58.201773 | 2026-03-13 02:11:58.201905 | TASK [add-fileserver : Add fileserver to inventory] 2026-03-13 02:11:58.288341 | localhost | changed 2026-03-13 02:11:58.297473 | 2026-03-13 02:11:58.297604 | TASK [add-fileserver : Add fileserver server to known hosts] 2026-03-13 02:11:58.749360 | localhost | changed 2026-03-13 02:11:58.790366 | 2026-03-13 02:11:58.790438 | PLAY [localhost] 2026-03-13 02:11:58.806052 | 2026-03-13 02:11:58.806138 | TASK [Generate bulk log download script] 2026-03-13 02:11:58.827055 | localhost | ok 2026-03-13 02:11:58.868620 | 2026-03-13 02:11:58.868785 | TASK [local-log-download : Check API endpoint is defined] 2026-03-13 02:11:58.908173 | localhost | ok: All assertions passed 2026-03-13 02:11:58.912557 | 2026-03-13 02:11:58.912616 | TASK [local-log-download : Create download script] 2026-03-13 02:11:59.373048 | localhost -> localhost | changed 2026-03-13 02:11:59.391279 | 2026-03-13 02:11:59.391408 | TASK [Register quick-download link] 2026-03-13 02:11:59.425895 | localhost | ok 2026-03-13 02:11:59.459068 | 2026-03-13 02:11:59.459142 | PLAY [logserver.rdoproject.org] 2026-03-13 02:11:59.473738 | 2026-03-13 02:11:59.473820 | TASK [Set zuul-log-path fact] 2026-03-13 02:11:59.492122 | logserver.rdoproject.org | ok 2026-03-13 02:11:59.503726 | 2026-03-13 02:11:59.503812 | TASK [set-zuul-log-path-fact : Set log path for a build] 2026-03-13 02:11:59.541851 | logserver.rdoproject.org | ok 2026-03-13 02:11:59.548885 | 2026-03-13 02:11:59.548965 | TASK [upload-logs : Create log directories] 2026-03-13 02:12:00.468212 | logserver.rdoproject.org | changed 2026-03-13 02:12:00.475207 | 2026-03-13 02:12:00.475343 | TASK [upload-logs : Ensure logs are readable before uploading] 2026-03-13 02:12:00.817261 | logserver.rdoproject.org -> localhost | ok: Runtime: 0:00:00.009929 2026-03-13 02:12:00.822089 | 2026-03-13 02:12:00.822152 | TASK [upload-logs : Upload logs to log server] 2026-03-13 02:12:01.615659 | logserver.rdoproject.org | Output suppressed because no_log was given 2026-03-13 02:12:01.621987 | 2026-03-13 02:12:01.622119 | LOOP [upload-logs : Compress console log and json output] 2026-03-13 02:12:01.690746 | logserver.rdoproject.org | skipping: Conditional result was False 2026-03-13 02:12:01.701923 | logserver.rdoproject.org | skipping: Conditional result was False 2026-03-13 02:12:01.710992 | 2026-03-13 02:12:01.711163 | LOOP [upload-logs : Upload compressed console log and json output] 2026-03-13 02:12:01.758202 | logserver.rdoproject.org | skipping: Conditional result was False 2026-03-13 02:12:01.758625 | 2026-03-13 02:12:01.761365 | logserver.rdoproject.org | skipping: Conditional result was False 2026-03-13 02:12:01.776750 | 2026-03-13 02:12:01.776914 | LOOP [upload-logs : Upload console log and json output]