2025-12-03 14:56:51.019868 | Job console starting...
2025-12-03 14:56:51.029542 | Updating repositories
2025-12-03 14:56:51.053607 | Preparing job workspace
2025-12-03 14:57:01.578025 | Running Ansible setup...
2025-12-03 14:57:05.415878 | PRE-RUN START: [trusted : review.rdoproject.org/config/playbooks/base-minimal/pre.yaml@master]
2025-12-03 14:57:06.017469 | 
2025-12-03 14:57:06.017632 | PLAY [localhost]
2025-12-03 14:57:06.028234 | 
2025-12-03 14:57:06.028326 | TASK [Gathering Facts]
2025-12-03 14:57:07.031866 | localhost | ok
2025-12-03 14:57:07.050086 | 
2025-12-03 14:57:07.050192 | TASK [log-inventory : Ensure Zuul Ansible directory exists]
2025-12-03 14:57:07.472067 | localhost -> localhost | changed
2025-12-03 14:57:07.477427 | 
2025-12-03 14:57:07.477496 | TASK [log-inventory : Copy ansible inventory to logs dir]
2025-12-03 14:57:08.382238 | localhost -> localhost | changed
2025-12-03 14:57:08.404255 | 
2025-12-03 14:57:08.404496 | TASK [Setup log path fact]
2025-12-03 14:57:08.430009 | localhost | ok
2025-12-03 14:57:08.454091 | 
2025-12-03 14:57:08.454213 | TASK [set-zuul-log-path-fact : Set log path for a build]
2025-12-03 14:57:08.494080 | localhost | ok
2025-12-03 14:57:08.504493 | 
2025-12-03 14:57:08.504579 | TASK [emit-job-header : Print job information]
2025-12-03 14:57:08.554473 | # Job Information
2025-12-03 14:57:08.554622 | Ansible Version: 2.15.12
2025-12-03 14:57:08.554657 | Job: vexxhost-hotstack-sno-2-bm
2025-12-03 14:57:08.554710 | Pipeline: github-experimental-trigger
2025-12-03 14:57:08.554736 | Executor: ze02.softwarefactory-project.io
2025-12-03 14:57:08.554760 | Triggered by: https://github.com/openstack-k8s-operators/hotstack/pull/235
2025-12-03 14:57:08.554784 | Log URL (when completed): https://logserver.rdoproject.org/f68/rdoproject.org/f689c7c4fae14d9e9a8e4ac1bc340a49/
2025-12-03 14:57:08.554809 | Event ID: 46de0420-d058-11f0-83a9-69f1a399778f
2025-12-03 14:57:08.559801 | 
2025-12-03 14:57:08.559886 | LOOP [emit-job-header : Print node information]
2025-12-03 14:57:08.662143 | localhost | ok:
2025-12-03 14:57:08.662316 | localhost | # Node Information
2025-12-03 14:57:08.662343 | localhost | Inventory Hostname: hotstack
2025-12-03 14:57:08.662365 | localhost | Hostname: np0005544261
2025-12-03 14:57:08.662385 | localhost | Username: zuul
2025-12-03 14:57:08.662405 | localhost | Distro: CentOS 9
2025-12-03 14:57:08.662422 | localhost | Provider: vexxhost-nodepool-tripleo
2025-12-03 14:57:08.662439 | localhost | Region: RegionOne
2025-12-03 14:57:08.662455 | localhost | Label: cloud-centos-9-stream-tripleo-vexxhost
2025-12-03 14:57:08.662471 | localhost | Product Name: OpenStack Nova
2025-12-03 14:57:08.662487 | localhost | Interface IP: 38.102.83.220
2025-12-03 14:57:08.706943 | 
2025-12-03 14:57:08.707125 | PLAY [all]
2025-12-03 14:57:08.721218 | 
2025-12-03 14:57:08.721364 | TASK [Gather network facts]
2025-12-03 14:57:09.381836 | hotstack | ok
2025-12-03 14:57:09.413482 | 
2025-12-03 14:57:09.413610 | TASK [include_role : start-zuul-console]
2025-12-03 14:57:09.437971 | hotstack | ok
2025-12-03 14:57:09.453477 | 
2025-12-03 14:57:09.453573 | TASK [start-zuul-console : Start zuul_console daemon.]
2025-12-03 14:57:09.936249 | hotstack | ok
2025-12-03 14:57:09.955193 | 
2025-12-03 14:57:09.955319 | TASK [include_role : add-build-sshkey]
2025-12-03 14:57:09.992384 | hotstack | ok
2025-12-03 14:57:10.019419 | 
2025-12-03 14:57:10.019504 | TASK [add-build-sshkey : Check to see if ssh key was already created for this build]
2025-12-03 14:57:10.299478 | hotstack -> localhost | ok
2025-12-03 14:57:10.313580 | 
2025-12-03 14:57:10.313768 | TASK [add-build-sshkey : Create a new key in workspace based on build UUID]
2025-12-03 14:57:10.349390 | hotstack | ok
2025-12-03 14:57:10.372989 | hotstack | included: /var/lib/zuul/builds/f689c7c4fae14d9e9a8e4ac1bc340a49/trusted/project_1/opendev.org/zuul/zuul-jobs/roles/add-build-sshkey/tasks/create-key-and-replace.yaml
2025-12-03 14:57:10.380641 | 
2025-12-03 14:57:10.380744 | TASK [add-build-sshkey : Create Temp SSH key]
2025-12-03 14:57:10.985390 | hotstack -> localhost | Generating public/private rsa key pair.
2025-12-03 14:57:10.985604 | hotstack -> localhost | Your identification has been saved in /var/lib/zuul/builds/f689c7c4fae14d9e9a8e4ac1bc340a49/work/f689c7c4fae14d9e9a8e4ac1bc340a49_id_rsa.
2025-12-03 14:57:10.985635 | hotstack -> localhost | Your public key has been saved in /var/lib/zuul/builds/f689c7c4fae14d9e9a8e4ac1bc340a49/work/f689c7c4fae14d9e9a8e4ac1bc340a49_id_rsa.pub.
2025-12-03 14:57:10.985657 | hotstack -> localhost | The key fingerprint is:
2025-12-03 14:57:10.985695 | hotstack -> localhost | SHA256:JocstHJ+QHG7+BAxh8xYGx/uOf5IFFpLkRC+IYC5jjA zuul-build-sshkey
2025-12-03 14:57:10.985715 | hotstack -> localhost | The key's randomart image is:
2025-12-03 14:57:10.985733 | hotstack -> localhost | +---[RSA 3072]----+
2025-12-03 14:57:10.985751 | hotstack -> localhost | |o. =O+=.         |
2025-12-03 14:57:10.985769 | hotstack -> localhost | |o...+@.+         |
2025-12-03 14:57:10.985787 | hotstack -> localhost | | .. B B          |
2025-12-03 14:57:10.985805 | hotstack -> localhost | |E  + % *         |
2025-12-03 14:57:10.985822 | hotstack -> localhost | |+.. X @ S        |
2025-12-03 14:57:10.985840 | hotstack -> localhost | |.. + B =         |
2025-12-03 14:57:10.985857 | hotstack -> localhost | |    . =          |
2025-12-03 14:57:10.985874 | hotstack -> localhost | |     o o         |
2025-12-03 14:57:10.985891 | hotstack -> localhost | |      . .        |
2025-12-03 14:57:10.985907 | hotstack -> localhost | +----[SHA256]-----+
2025-12-03 14:57:10.985952 | hotstack -> localhost | ok: Runtime: 0:00:00.220324
2025-12-03 14:57:10.991630 | 
2025-12-03 14:57:10.991706 | TASK [add-build-sshkey : Remote setup ssh keys (linux)]
2025-12-03 14:57:11.014254 | hotstack | ok
2025-12-03 14:57:11.024096 | hotstack | included: /var/lib/zuul/builds/f689c7c4fae14d9e9a8e4ac1bc340a49/trusted/project_1/opendev.org/zuul/zuul-jobs/roles/add-build-sshkey/tasks/remote-linux.yaml
2025-12-03 14:57:11.031854 | 
2025-12-03 14:57:11.031916 | TASK [add-build-sshkey : Remove previously added zuul-build-sshkey]
2025-12-03 14:57:11.046179 | hotstack | skipping: Conditional result was False
2025-12-03 14:57:11.052932 | 
2025-12-03 14:57:11.053001 | TASK [add-build-sshkey : Enable access via build key on all nodes]
2025-12-03 14:57:11.513828 | hotstack | changed
2025-12-03 14:57:11.524494 | 
2025-12-03 14:57:11.524650 | TASK [add-build-sshkey : Make sure user has a .ssh]
2025-12-03 14:57:11.816447 | hotstack | ok
2025-12-03 14:57:11.827236 | 
2025-12-03 14:57:11.827371 | TASK [add-build-sshkey : Install build private key as SSH key on all nodes]
2025-12-03 14:57:12.912932 | hotstack | changed
2025-12-03 14:57:12.924360 | 
2025-12-03 14:57:12.924527 | TASK [add-build-sshkey : Install build public key as SSH key on all nodes]
2025-12-03 14:57:13.900320 | hotstack | changed
2025-12-03 14:57:13.911624 | 
2025-12-03 14:57:13.911796 | TASK [add-build-sshkey : Remote setup ssh keys (windows)]
2025-12-03 14:57:13.939379 | hotstack | skipping: Conditional result was False
2025-12-03 14:57:13.952431 | 
2025-12-03 14:57:13.952568 | TASK [remove-zuul-sshkey : Remove master key from local agent]
2025-12-03 14:57:14.400155 | hotstack -> localhost | changed
2025-12-03 14:57:14.422543 | 
2025-12-03 14:57:14.422753 | TASK [add-build-sshkey : Add back temp key]
2025-12-03 14:57:14.755575 | hotstack -> localhost | Identity added: /var/lib/zuul/builds/f689c7c4fae14d9e9a8e4ac1bc340a49/work/f689c7c4fae14d9e9a8e4ac1bc340a49_id_rsa (zuul-build-sshkey)
2025-12-03 14:57:14.755862 | hotstack -> localhost | ok: Runtime: 0:00:00.008261
2025-12-03 14:57:14.761744 | 
2025-12-03 14:57:14.761808 | TASK [add-build-sshkey : Verify we can still SSH to all nodes]
2025-12-03 14:57:15.187871 | hotstack | ok
2025-12-03 14:57:15.192581 | 
2025-12-03 14:57:15.192643 | TASK [add-build-sshkey : Verify we can still SSH to all nodes (windows)]
2025-12-03 14:57:15.227343 | hotstack | skipping: Conditional result was False
2025-12-03 14:57:15.237212 | 
2025-12-03 14:57:15.237277 | TASK [include_role : validate-host]
2025-12-03 14:57:15.266003 | hotstack | ok
2025-12-03 14:57:15.285584 | 
2025-12-03 14:57:15.285657 | TASK [validate-host : Define zuul_info_dir fact]
2025-12-03 14:57:15.324719 | hotstack | ok
2025-12-03 14:57:15.329479 | 
2025-12-03 14:57:15.329545 | TASK [validate-host : Ensure Zuul Ansible directory exists]
2025-12-03 14:57:15.608644 | hotstack -> localhost | ok
2025-12-03 14:57:15.622014 | 
2025-12-03 14:57:15.622325 | TASK [validate-host : Collect information about the host]
2025-12-03 14:57:16.442556 | hotstack | ok
2025-12-03 14:57:16.461134 | 
2025-12-03 14:57:16.461272 | TASK [validate-host : Sanitize hostname]
2025-12-03 14:57:16.540273 | hotstack | ok
2025-12-03 14:57:16.550643 | 
2025-12-03 14:57:16.550811 | TASK [validate-host : Write out all ansible variables/facts known for each host]
2025-12-03 14:57:17.108519 | hotstack -> localhost | changed
2025-12-03 14:57:17.120585 | 
2025-12-03 14:57:17.120754 | TASK [validate-host : Collect information about zuul worker]
2025-12-03 14:57:17.575153 | hotstack | ok
2025-12-03 14:57:17.585331 | 
2025-12-03 14:57:17.585466 | TASK [validate-host : Write out all zuul information for each host]
2025-12-03 14:57:18.173964 | hotstack -> localhost | changed
2025-12-03 14:57:18.201894 | 
2025-12-03 14:57:18.202118 | TASK [include_role : prepare-workspace-openshift]
2025-12-03 14:57:18.228960 | hotstack | skipping: Conditional result was False
2025-12-03 14:57:18.238854 | 
2025-12-03 14:57:18.238986 | TASK [include_role : remove-zuul-sshkey]
2025-12-03 14:57:18.264130 | hotstack | skipping: Conditional result was False
2025-12-03 14:57:18.270968 | 
2025-12-03 14:57:18.271059 | LOOP [ensure-output-dirs : Empty Zuul Output directories by removing them]
2025-12-03 14:57:18.576967 | hotstack | ok: "logs"
2025-12-03 14:57:18.577401 | hotstack | ok: All items complete
2025-12-03 14:57:18.577463 | 
2025-12-03 14:57:18.814432 | hotstack | ok: "artifacts"
2025-12-03 14:57:19.050429 | hotstack | ok: "docs"
2025-12-03 14:57:19.069552 | 
2025-12-03 14:57:19.069732 | LOOP [ensure-output-dirs : Ensure Zuul Output directories exist]
2025-12-03 14:57:19.354326 | hotstack | changed: "logs"
2025-12-03 14:57:19.609492 | hotstack | changed: "artifacts"
2025-12-03 14:57:19.859171 | hotstack | changed: "docs"
2025-12-03 14:57:19.916560 | 
2025-12-03 14:57:19.916728 | PLAY RECAP
2025-12-03 14:57:19.916792 | hotstack | ok: 22 changed: 9 unreachable: 0 failed: 0 skipped: 5 rescued: 0 ignored: 0
2025-12-03 14:57:19.916828 | localhost | ok: 6 changed: 2 unreachable: 0 failed: 0 skipped: 0 rescued: 0 ignored: 0
2025-12-03 14:57:19.916854 | 
2025-12-03 14:57:20.054071 | PRE-RUN END RESULT_NORMAL: [trusted : review.rdoproject.org/config/playbooks/base-minimal/pre.yaml@master]
2025-12-03 14:57:20.060339 | PRE-RUN START: [trusted : review.rdoproject.org/config/playbooks/hotstack/pre_stage1.yaml@master]
2025-12-03 14:57:20.662478 | 
2025-12-03 14:57:20.662586 | PLAY [Create necessary files to deploy hotstack scenario]
2025-12-03 14:57:20.683435 | 
2025-12-03 14:57:20.683519 | TASK [prepare-workspace : Start zuul_console daemon.]
2025-12-03 14:57:21.116897 | hotstack | ok
2025-12-03 14:57:21.122945 | 
2025-12-03 14:57:21.123020 | TASK [prepare-workspace : Synchronize src repos to workspace directory.]
2025-12-03 14:57:24.003921 | hotstack | Output suppressed because no_log was given
2025-12-03 14:57:24.027337 | 
2025-12-03 14:57:24.027537 | LOOP [add-authorized-keys : Enable access via build key on all nodes]
2025-12-03 14:57:24.538391 | hotstack | changed:
2025-12-03 14:57:24.539244 | hotstack | {
2025-12-03 14:57:24.539328 | hotstack |   "public_key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCINPOs6XsfMVQMqjmiduT3Bxhogy7MbAFOWlR+9LlKkTW1kV6glsCXY5svtggd2zRtc40UWalHaj9jC+yEkZdI35h2PkK+Sww52u4nczqgNpIFNt7Dm2xtYKbTolAV9jSeNkRlGMCq9EBzQlwiV0HkHx1CdNR/zzXbyNZLmOybFkXqR150H01kUhdv0v6AMWTDZjMEWO/PaWCOgzZhH9ztXjy+rwa7pdaI4+0Pb9Kib+d8/IrgNsLB/sCazqD+9/COI6AHqzTDS1aU9/gMQrUN1ronzMEQhzzOuoI8WeLqU++j+0dT2UWVPlQqoln+RYu7dUn/DSZDrsYenudT5+P"
2025-12-03 14:57:24.539376 | hotstack | }
2025-12-03 14:57:24.572830 | 
2025-12-03 14:57:24.573211 | TASK [Assert cloud name and cloud secret]
2025-12-03 14:57:24.623637 | hotstack | Output suppressed because no_log was given
2025-12-03 14:57:24.635249 | 
2025-12-03 14:57:24.635405 | TASK [Ensure openstack release repo is installed]
2025-12-03 14:57:29.874705 | hotstack | changed
2025-12-03 14:57:29.886472 | 
2025-12-03 14:57:29.886610 | TASK [Install openstack client packages]
2025-12-03 14:57:54.065953 | hotstack | changed
2025-12-03 14:57:54.078950 | 
2025-12-03 14:57:54.079096 | TASK [Ensure HotStack work dir]
2025-12-03 14:57:54.529419 | hotstack | changed
2025-12-03 14:57:54.540764 | 
2025-12-03 14:57:54.540899 | LOOP [Ensure etc directories]
2025-12-03 14:57:54.848483 | hotstack | changed: "/etc/openstack"
2025-12-03 14:57:54.848973 | hotstack | changed: All items complete
2025-12-03 14:57:54.849046 | 
2025-12-03 14:57:55.089036 | hotstack | changed: "/etc/hotstack"
2025-12-03 14:57:55.113358 | 
2025-12-03 14:57:55.113541 | TASK [Ensure .ssh dir exists]
2025-12-03 14:57:55.357793 | hotstack | Output suppressed because no_log was given
2025-12-03 14:57:55.370565 | 
2025-12-03 14:57:55.370748 | TASK [Ensure ssh keypair exists]
2025-12-03 14:58:05.936388 | hotstack | Output suppressed because no_log was given
2025-12-03 14:58:05.948250 | 
2025-12-03 14:58:05.948380 | TASK [Generate CA cert file]
2025-12-03 14:58:05.985582 | hotstack | skipping: Conditional result was False
2025-12-03 14:58:05.997111 | 
2025-12-03 14:58:05.997238 | TASK [Update CA certs]
2025-12-03 14:58:06.545846 | hotstack | skipping: Conditional result was False
2025-12-03 14:58:06.559572 | 
2025-12-03 14:58:06.559771 | TASK [Set _auth_type fact]
2025-12-03 14:58:06.617874 | hotstack | Output suppressed because no_log was given
2025-12-03 14:58:06.628317 | 
2025-12-03 14:58:06.628485 | TASK [Write clouds.yaml - user/password]
2025-12-03 14:58:07.957810 | hotstack | Output suppressed because no_log was given
2025-12-03 14:58:07.974305 | 
2025-12-03 14:58:07.977495 | TASK [Create application credential for sushy-emulator]
2025-12-03 14:58:10.011652 | hotstack | Output suppressed because no_log was given
2025-12-03 14:58:10.019056 | 
2025-12-03 14:58:10.019160 | TASK [Read application credential from file]
2025-12-03 14:58:10.396611 | hotstack | Output suppressed because no_log was given
2025-12-03 14:58:10.401561 | 
2025-12-03 14:58:10.401627 | TASK [Delete temporary application credential file]
2025-12-03 14:58:10.660402 | hotstack | Output suppressed because no_log was given
2025-12-03 14:58:10.665833 | 
2025-12-03 14:58:10.665911 | TASK [Assert application credential id and secret]
2025-12-03 14:58:10.750534 | hotstack | Output suppressed because no_log was given
2025-12-03 14:58:10.757191 | 
2025-12-03 14:58:10.757272 | TASK [Write new clouds.yaml with application credentials]
2025-12-03 14:58:11.625835 | hotstack | Output suppressed because no_log was given
2025-12-03 14:58:11.639230 | 
2025-12-03 14:58:11.639377 | TASK [Write hotstack cloud-secret.yaml]
2025-12-03 14:58:12.575789 | hotstack | Output suppressed because no_log was given
2025-12-03 14:58:12.586469 | 
2025-12-03 14:58:12.586597 | TASK [Write clouds.yaml with application credentials]
2025-12-03 14:58:12.625284 | 
2025-12-03 14:58:12.625549 | TASK [Write hotstack cloud-secret.yaml]
2025-12-03 14:58:12.719062 | 
2025-12-03 14:58:12.719178 | PLAY RECAP
2025-12-03 14:58:12.719217 | hotstack | ok: 18 changed: 11 unreachable: 0 failed: 0 skipped: 4 rescued: 0 ignored: 0
2025-12-03 14:58:12.719237 | 
2025-12-03 14:58:12.809466 | PRE-RUN END RESULT_NORMAL: [trusted : review.rdoproject.org/config/playbooks/hotstack/pre_stage1.yaml@master]
2025-12-03 14:58:12.812982 | PRE-RUN START: [trusted : review.rdoproject.org/config/playbooks/hotstack/pre_stage2.yaml@master]
2025-12-03 14:58:13.424638 | 
2025-12-03 14:58:13.424777 | PLAY [Create hotstack scenario]
2025-12-03 14:58:13.445609 | 
2025-12-03 14:58:13.445719 | TASK [Assert scenario is defined]
2025-12-03 14:58:13.531326 | hotstack | ok: All assertions passed
2025-12-03 14:58:13.543620 | 
2025-12-03 14:58:13.543801 | TASK [Fetch bootstrap vars from remote to zuul executor]
2025-12-03 14:58:14.031272 | hotstack | Output suppressed because no_log was given
2025-12-03 14:58:14.043260 | 
2025-12-03 14:58:14.043425 | TASK [Load bootstrap vars with template resolution]
2025-12-03 14:58:14.093347 | hotstack | Output suppressed because no_log was given
2025-12-03 14:58:14.105152 | 
2025-12-03 14:58:14.105290 | TASK [Fetch automation vars from remote to zuul executor]
2025-12-03 14:58:14.509839 | hotstack | Output suppressed because no_log was given
2025-12-03 14:58:14.520297 | 
2025-12-03 14:58:14.520428 | TASK [Load automation vars with template resolution]
2025-12-03 14:58:14.609219 | hotstack | Output suppressed because no_log was given
2025-12-03 14:58:14.619767 | 
2025-12-03 14:58:14.619896 | TASK [Slurp zuul-build-sshkey from file]
2025-12-03 14:58:15.003847 | hotstack | Output suppressed because no_log was given
2025-12-03 14:58:15.015861 | 
2025-12-03 14:58:15.015993 | TASK [Set up infrastructure (Heat stack)]
2025-12-03 14:58:15.043957 | hotstack | Output suppressed because no_log was given
2025-12-03 14:58:15.102145 | 
2025-12-03 14:58:15.102300 | TASK [dataplane_ssh_keys : Ensure Dataplane ssh keypair exists]
2025-12-03 14:58:15.700428 | hotstack | changed
2025-12-03 14:58:15.711984 | 
2025-12-03 14:58:15.712130 | TASK [dataplane_ssh_keys : Ensure Nova migration ssh keypair exists]
2025-12-03 14:58:16.127104 | hotstack | changed
2025-12-03 14:58:16.139373 | 
2025-12-03 14:58:16.139509 | TASK [dataplane_ssh_keys : Set SSH public key facts]
2025-12-03 14:58:16.189130 | hotstack | ok
2025-12-03 14:58:16.201082 | 
2025-12-03 14:58:16.201215 | TASK [dataplane_ssh_keys : Store SSH public key facts in file]
2025-12-03 14:58:17.218438 | hotstack | changed
2025-12-03 14:58:17.240128 | 
2025-12-03 14:58:17.240269 | TASK [heat_stack : Assert config is defined]
2025-12-03 14:58:17.360008 | hotstack | ok: All assertions passed
2025-12-03 14:58:17.371074 | 
2025-12-03 14:58:17.371225 | TASK [heat_stack : Get latest snapset and update stack parameters]
2025-12-03 14:58:17.430354 | hotstack | skipping: Conditional result was False
2025-12-03 14:58:17.444069 | 
2025-12-03 14:58:17.444219 | TASK [heat_stack : Debug latest snapset information]
2025-12-03 14:58:17.491559 | hotstack | skipping: Conditional result was False
2025-12-03 14:58:17.504490 | 
2025-12-03 14:58:17.504632 | TASK [heat_stack : Create stack]
2025-12-03 15:19:24.444338 | hotstack | changed
2025-12-03 15:19:24.465480 | 
2025-12-03 15:19:24.465725 | TASK [heat_stack : Wait for stack to complete]
2025-12-03 15:19:26.537508 | hotstack | ok
2025-12-03 15:19:26.551272 | 
2025-12-03 15:19:26.551412 | TASK [heat_stack : Debug - stack_info]
2025-12-03 15:19:26.613940 | hotstack | skipping: Conditional result was False
2025-12-03 15:19:26.632778 | 
2025-12-03 15:19:26.632970 | TASK [heat_stack : Stop if stack not complete]
2025-12-03 15:19:26.663211 | hotstack | skipping: Conditional result was False
2025-12-03 15:19:26.676842 | 
2025-12-03 15:19:26.677064 | TASK [heat_stack : Get stack outputs]
2025-12-03 15:19:28.511222 | hotstack | changed
2025-12-03 15:19:28.528519 | 
2025-12-03 15:19:28.528659 | TASK [heat_stack : Set stack outputs facts]
2025-12-03 15:19:28.579097 | hotstack | ok
2025-12-03 15:19:28.595898 | 
2025-12-03 15:19:28.596089 | TASK [heat_stack : Store stack outputs in file]
2025-12-03 15:19:29.468052 | hotstack | changed
2025-12-03 15:19:29.494804 | 
2025-12-03 15:19:29.495007 | TASK [Fetch stack outputs from remote to zuul executor]
2025-12-03 15:19:29.850500 | hotstack | Output suppressed because no_log was given
2025-12-03 15:19:29.862078 | 
2025-12-03 15:19:29.862227 | TASK [Load stack output vars with template resolution]
2025-12-03 15:19:30.033112 | hotstack | Output suppressed because no_log was given
2025-12-03 15:19:30.040364 | 
2025-12-03 15:19:30.040508 | TASK [Fetch cloud secrets from remote to zuul executor]
2025-12-03 15:19:30.419049 | hotstack | Output suppressed because no_log was given
2025-12-03 15:19:30.424927 | 
2025-12-03 15:19:30.425008 | TASK [Load hotstack cloud secret vars with template resolution]
2025-12-03 15:19:30.467531 | hotstack | Output suppressed because no_log was given
2025-12-03 15:19:30.474545 | 
2025-12-03 15:19:30.474644 | TASK [Add Hotstack controller-0 to ansible inventory]
2025-12-03 15:19:30.508204 | hotstack | Output suppressed because no_log was given
2025-12-03 15:19:30.519872 | 
2025-12-03 15:19:30.519988 | TASK [Wait for Hotstack controller-0 to be ready]
2025-12-03 15:19:36.828957 | hotstack -> 38.102.83.30 | Output suppressed because no_log was given
2025-12-03 15:19:36.847070 | 
2025-12-03 15:19:36.847243 | TASK [Start Zuul console on Hotstack controller-0]
2025-12-03 15:19:36.887850 | hotstack | Output suppressed because no_log was given
2025-12-03 15:19:36.933068 | 
2025-12-03 15:19:36.933230 | TASK [start-zuul-console : Start zuul_console daemon.]
2025-12-03 15:19:37.418482 | hotstack -> 38.102.83.30 | ok
2025-12-03 15:19:37.443516 | 
2025-12-03 15:19:37.443739 | TASK [Prepare the controller node]
2025-12-03 15:19:37.475187 | hotstack | Output suppressed because no_log was given
2025-12-03 15:19:37.554489 | 
2025-12-03 15:19:37.554639 | TASK [controller : Assert config is defined]
2025-12-03 15:19:37.649714 | hotstack | ok: All assertions passed
2025-12-03 15:19:37.659421 | 
2025-12-03 15:19:37.659534 | TASK [controller : Add controller-0 to the Ansible inventory]
2025-12-03 15:19:37.692744 | hotstack | ok
2025-12-03 15:19:37.700825 | 
2025-12-03 15:19:37.700912 | TASK [controller : Read Dataplane SSH private key content]
2025-12-03 15:19:37.949554 | hotstack | ok
2025-12-03 15:19:37.962496 | 
2025-12-03 15:19:37.962633 | TASK [controller : Read Nova migration SSH private key content]
2025-12-03 15:19:38.236106 | hotstack | ok
2025-12-03 15:19:38.252022 | 
2025-12-03 15:19:38.252312 | TASK [controller : Wait for controller-0 to be ready]
2025-12-03 15:19:39.097041 | hotstack -> 38.102.83.30 | ok
2025-12-03 15:19:39.109334 | 
2025-12-03 15:19:39.109470 | LOOP [controller : Ensure directories exists]
2025-12-03 15:19:39.949628 | hotstack -> 38.102.83.30 | changed: "/home/zuul/data"
2025-12-03 15:19:40.545567 | hotstack -> 38.102.83.30 | changed: "/home/zuul/bin"
2025-12-03 15:19:40.566908 | 
2025-12-03 15:19:40.567355 | LOOP [controller : Add common functions to the bin_dir]
2025-12-03 15:19:41.906741 | hotstack -> 38.102.83.30 | changed: "bin/hotstack-common-funcs.sh"
2025-12-03 15:19:41.907280 | hotstack -> 38.102.83.30 | changed: All items complete
2025-12-03 15:19:41.907347 | 
2025-12-03 15:19:41.924702 | 
2025-12-03 15:19:41.924950 | LOOP [controller : Add bin utilities to the bin_dir]
2025-12-03 15:19:43.476632 | hotstack -> 38.102.83.30 | changed: "bin/hotstack-approve-install-plan"
2025-12-03 15:19:44.484010 | hotstack -> 38.102.83.30 | changed: "bin/hotstack-collect-tempest-logs"
2025-12-03 15:19:45.232758 | hotstack -> 38.102.83.30 | changed: "bin/hotstack-leader-election-tune"
2025-12-03 15:19:45.960000 | hotstack -> 38.102.83.30 | changed: "bin/hotstack-nova-discover-hosts"
2025-12-03 15:19:46.663475 | hotstack -> 38.102.83.30 | changed: "bin/hotstack-snapset"
2025-12-03 15:19:47.386787 | hotstack -> 38.102.83.30 | changed: "bin/hotstack-openstack-version-patch"
2025-12-03 15:19:48.150170 | hotstack -> 38.102.83.30 | changed: "bin/hotstack-wait-for-bmh"
2025-12-03 15:19:48.172811 | 
2025-12-03 15:19:48.173028 | TASK [controller : Write ansible inventory to file on controller-0]
2025-12-03 15:19:48.242573 | hotstack | skipping: Conditional result was False
2025-12-03 15:19:48.257787 | 
2025-12-03 15:19:48.257997 | TASK [controller : Copy Dataplane ssh private key to controller]
2025-12-03 15:19:49.084600 | hotstack -> 38.102.83.30 | changed
2025-12-03 15:19:49.092509 | 
2025-12-03 15:19:49.092623 | TASK [controller : Copy Dataplane ssh public key to controller]
2025-12-03 15:19:49.971611 | hotstack -> 38.102.83.30 | changed
2025-12-03 15:19:49.988629 | 
2025-12-03 15:19:49.989066 | TASK [controller : Copy Nova migrate ssh private key to controller]
2025-12-03 15:19:50.750897 | hotstack -> 38.102.83.30 | changed
2025-12-03 15:19:50.757558 | 
2025-12-03 15:19:50.757637 | TASK [controller : Copy Nova migrate ssh public key to controller]
2025-12-03 15:19:51.597170 | hotstack -> 38.102.83.30 | changed
2025-12-03 15:19:51.670852 | 
2025-12-03 15:19:51.671050 | TASK [controller : Copy dataplane authorized keys to controller]
2025-12-03 15:19:52.416831 | hotstack -> 38.102.83.30 | changed
2025-12-03 15:19:52.428811 | 
2025-12-03 15:19:52.428928 | TASK [controller : Ensure cloud config directory exists]
2025-12-03 15:19:52.784489 | hotstack -> 38.102.83.30 | changed
2025-12-03 15:19:52.794311 | 
2025-12-03 15:19:52.794400 | TASK [controller : Template the clouds.yaml]
2025-12-03 15:19:53.627588 | hotstack -> 38.102.83.30 | Output suppressed because no_log was given
2025-12-03 15:19:53.650573 | 
2025-12-03 15:19:53.650809 | TASK [controller : Write CA certificate to file]
2025-12-03 15:19:54.526479 | hotstack -> 38.102.83.30 | Output suppressed because no_log was given
2025-12-03 15:19:54.539819 | 
2025-12-03 15:19:54.539984 | TASK [controller : Write Ironic nodes YAML]
2025-12-03 15:19:55.375683 | hotstack -> 38.102.83.30 | changed
2025-12-03 15:19:55.386875 | 
2025-12-03 15:19:55.386938 | TASK [Install Openshift]
2025-12-03 15:19:55.407312 | hotstack | Output suppressed because no_log was given
2025-12-03 15:19:55.446767 | 
2025-12-03 15:19:55.446863 | TASK [ocp_agent_installer : Assert config is defined]
2025-12-03 15:19:55.502484 | hotstack -> 38.102.83.30 | ok: All assertions passed
2025-12-03 15:19:55.511593 | 
2025-12-03 15:19:55.511711 | TASK [ocp_agent_installer : Install using agent installer]
2025-12-03 15:19:55.568158 | hotstack | ok
2025-12-03 15:19:55.585195 | hotstack | included: /var/lib/zuul/builds/f689c7c4fae14d9e9a8e4ac1bc340a49/trusted/project_3/github.com/openstack-k8s-operators/hotstack/roles/ocp_agent_installer/tasks/_install.yml
2025-12-03 15:19:55.594042 | 
2025-12-03 15:19:55.594132 | TASK [ocp_agent_installer : Assert config is defined]
2025-12-03 15:19:55.650089 | hotstack -> 38.102.83.30 | ok: All assertions passed
2025-12-03 15:19:55.656787 | 
2025-12-03 15:19:55.656853 | LOOP [ocp_agent_installer : Ensure directory exists]
2025-12-03 15:19:56.021356 | hotstack -> 38.102.83.30 | ok: "/home/zuul/bin"
2025-12-03 15:19:56.021812 | hotstack -> 38.102.83.30 | changed: All items complete
2025-12-03 15:19:56.021853 | 
2025-12-03 15:19:56.326341 | hotstack -> 38.102.83.30 | changed: "/home/zuul/.kube"
2025-12-03 15:19:56.631937 | hotstack -> 38.102.83.30 | changed: "/home/zuul/ocp-cluster"
2025-12-03 15:19:56.951606 | hotstack -> 38.102.83.30 | changed: "/home/zuul/ocp-cluster/openshift"
2025-12-03 15:19:57.268878 | hotstack -> 38.102.83.30 | changed: "/home/zuul/agent-installer"
2025-12-03 15:19:57.573031 | hotstack -> 38.102.83.30 | changed: "/home/zuul/cluster-custom-config/"
2025-12-03 15:19:57.874144 | hotstack -> 38.102.83.30 | changed: "/home/zuul/cluster-custom-config//butane"
2025-12-03 15:19:58.180397 | hotstack -> 38.102.83.30 | changed: "/home/zuul/cluster-custom-config//machine-configs"
2025-12-03 15:19:58.483012 | hotstack -> 38.102.83.30 | changed: "/home/zuul/cluster-custom-config//config-assets"
2025-12-03 15:19:58.506760 | 
2025-12-03 15:19:58.506927 | TASK [ocp_agent_installer : Install package requirements for agent installer]
2025-12-03 15:20:21.632120 | hotstack -> 38.102.83.30 | ok: Nothing to do
2025-12-03 15:20:21.654639 | 
2025-12-03 15:20:21.654901 | TASK [ocp_agent_installer : Run tasks/install_client.yml]
2025-12-03 15:20:21.684398 | hotstack | ok
2025-12-03 15:20:21.709007 | hotstack | included: /var/lib/zuul/builds/f689c7c4fae14d9e9a8e4ac1bc340a49/trusted/project_3/github.com/openstack-k8s-operators/hotstack/roles/ocp_agent_installer/tasks/install_client.yml
2025-12-03 15:20:21.728035 | 
2025-12-03 15:20:21.728170 | TASK [ocp_agent_installer : Download the client]
2025-12-03 15:20:24.215780 | hotstack -> 38.102.83.30 | changed: OK (73347738 bytes)
2025-12-03 15:20:24.223770 | 
2025-12-03 15:20:24.223839 | TASK [ocp_agent_installer : Extract client to /home/zuul/bin]
2025-12-03 15:20:28.943141 | hotstack -> 38.102.83.30 | changed
2025-12-03 15:20:28.957111 | 
2025-12-03 15:20:28.957246 | TASK [ocp_agent_installer : Configure bash completion]
2025-12-03 15:20:30.492062 | hotstack -> 38.102.83.30 | changed
2025-12-03 15:20:30.558879 | 
2025-12-03 15:20:30.559030 | TASK [ocp_agent_installer : Run tasks/install_installer.yml]
2025-12-03 15:20:30.590813 | hotstack | ok
2025-12-03 15:20:30.600906 | hotstack | included: /var/lib/zuul/builds/f689c7c4fae14d9e9a8e4ac1bc340a49/trusted/project_3/github.com/openstack-k8s-operators/hotstack/roles/ocp_agent_installer/tasks/install_installer.yml
2025-12-03 15:20:30.618190 | 
2025-12-03 15:20:30.618277 | TASK [ocp_agent_installer : Download the installer]
2025-12-03 15:20:38.804484 | hotstack -> 38.102.83.30 | changed: OK (437703087 bytes)
2025-12-03 15:20:38.811063 | 
2025-12-03 15:20:38.811137 | TASK [ocp_agent_installer : Extract installer to /home/zuul/bin]
2025-12-03 15:20:55.315517 | hotstack -> 38.102.83.30 | changed
2025-12-03 15:20:55.335761 | 
2025-12-03 15:20:55.335966 | TASK [ocp_agent_installer : Write OCP install-config.yaml]
2025-12-03 15:20:58.806988 | hotstack -> 38.102.83.30 | changed
2025-12-03 15:20:58.813881 | 
2025-12-03 15:20:58.813949 | TASK [ocp_agent_installer : Replace pull secret in install-config.yaml]
2025-12-03 15:20:59.304414 | hotstack -> 38.102.83.30 | Output suppressed because no_log was given
2025-12-03 15:20:59.310967 | 
2025-12-03 15:20:59.311057 | TASK [ocp_agent_installer : Write OCP agent-config.yaml]
2025-12-03 15:21:00.170648 | hotstack -> 38.102.83.30 | changed
2025-12-03 15:21:00.177114 | 
2025-12-03 15:21:00.177179 | TASK [ocp_agent_installer : Include tasks to generate machine configs]
2025-12-03 15:21:00.207369 | hotstack | ok
2025-12-03 15:21:00.221324 | hotstack | included: /var/lib/zuul/builds/f689c7c4fae14d9e9a8e4ac1bc340a49/trusted/project_3/github.com/openstack-k8s-operators/hotstack/roles/ocp_agent_installer/tasks/machine_configs.yml
2025-12-03 15:21:00.244461 | 
2025-12-03 15:21:00.244535 | LOOP [ocp_agent_installer : Template butane config for net.ifnames]
2025-12-03 15:21:01.180289 | hotstack -> 38.102.83.30 | changed: "master"
2025-12-03 15:21:01.214786 | 
2025-12-03 15:21:01.214910 | LOOP [ocp_agent_installer : Generate MachineConfig for iscsi]
2025-12-03 15:21:02.142226 | hotstack -> 38.102.83.30 | changed
2025-12-03 15:21:02.142577 | hotstack -> 38.102.83.30 | changed: All items complete
2025-12-03 15:21:02.142619 | 
2025-12-03 15:21:02.160936 | 
2025-12-03 15:21:02.161047 | LOOP [ocp_agent_installer : Template butane config for iscsi]
2025-12-03 15:21:02.936693 | hotstack -> 38.102.83.30 | changed: "master"
2025-12-03 15:21:02.956252 | 
2025-12-03 15:21:02.956396 | LOOP [ocp_agent_installer : Generate MachineConfig for iscsi]
2025-12-03 15:21:03.364368 | hotstack -> 38.102.83.30 | changed
2025-12-03 15:21:03.364645 | hotstack -> 38.102.83.30 | changed: All items complete
2025-12-03 15:21:03.364699 | 
2025-12-03 15:21:03.378085 | 
2025-12-03 15:21:03.378228 | LOOP [ocp_agent_installer : Template butane config for multipath]
2025-12-03 15:21:04.245317 | hotstack -> 38.102.83.30 | changed: "master"
2025-12-03 15:21:04.267272 | 
2025-12-03 15:21:04.267485 | LOOP [ocp_agent_installer : Generate MachineConfig for multipath]
2025-12-03 15:21:04.750890 | hotstack -> 38.102.83.30 | changed
2025-12-03 15:21:04.751144 | hotstack -> 38.102.83.30 | changed: All items complete
2025-12-03 15:21:04.751171 | 
2025-12-03 15:21:04.763438 | 
2025-12-03 15:21:04.763508 | LOOP [ocp_agent_installer : Template butane config for LVM cinder-volumes]
2025-12-03 15:21:05.584974 | hotstack -> 38.102.83.30 | changed: "master"
2025-12-03 15:21:05.602498 | 
2025-12-03 15:21:05.602729 | LOOP [ocp_agent_installer : Generate MachineConfig for multipath]
2025-12-03 15:21:06.025149 | hotstack -> 38.102.83.30 | changed
2025-12-03 15:21:06.025604 | hotstack -> 38.102.83.30 | changed: All items complete
2025-12-03 15:21:06.025694 | 
2025-12-03 15:21:06.042223 | 
2025-12-03 15:21:06.042370 | TASK [ocp_agent_installer : Include tasks to generate config assets]
2025-12-03 15:21:06.081024 | hotstack | ok
2025-12-03 15:21:06.101087 | hotstack | included: /var/lib/zuul/builds/f689c7c4fae14d9e9a8e4ac1bc340a49/trusted/project_3/github.com/openstack-k8s-operators/hotstack/roles/ocp_agent_installer/tasks/config_assets.yml
2025-12-03 15:21:06.133403 | 
2025-12-03 15:21:06.133499 | TASK [ocp_agent_installer : Template ovn-k8s customization]
2025-12-03 15:21:06.935462 | hotstack -> 38.102.83.30 | changed
2025-12-03 15:21:06.944368 | 
2025-12-03 15:21:06.944476 | TASK [ocp_agent_installer : Copy Etcd customization]
2025-12-03 15:21:07.734494 | hotstack -> 38.102.83.30 | changed
2025-12-03 15:21:07.742797 | 
2025-12-03 15:21:07.742883 | TASK [ocp_agent_installer : Template ImageContentSourcePolicy customization]
2025-12-03 15:21:07.809441 | hotstack | skipping: Conditional result was False
2025-12-03 15:21:07.817751 | 
2025-12-03 15:21:07.817878 | TASK [ocp_agent_installer : Initialize _ocp_additional_trusted_ca_map fact]
2025-12-03 15:21:07.878456 | hotstack | skipping: Conditional result was False
2025-12-03 15:21:07.886734 | 
2025-12-03 15:21:07.886833 | LOOP [ocp_agent_installer : Append to _ocp_additional_trusted_ca_map fact]
2025-12-03 15:21:07.936648 | hotstack | skipping: Conditional result was False
2025-12-03 15:21:07.953104 | 
2025-12-03 15:21:07.953269 | TASK [ocp_agent_installer : Template additional CA config map]
2025-12-03 15:21:07.999700 | hotstack | skipping: Conditional result was False
2025-12-03 15:21:08.009305 | 
2025-12-03 15:21:08.009474 | TASK [ocp_agent_installer : Template Image config (additionalTrustedCA and/or insecureRegistries)]
2025-12-03 15:21:08.056224 | hotstack | skipping: Conditional result was False
2025-12-03 15:21:08.065219 | 
2025-12-03 15:21:08.065359 | TASK [ocp_agent_installer : Copy machine_configs to manifests dir]
2025-12-03 15:21:08.530931 | hotstack -> 38.102.83.30 | changed
2025-12-03 15:21:08.545867 | 
2025-12-03 15:21:08.546060 | TASK [ocp_agent_installer : Copy config assets to manifests dir]
2025-12-03 15:21:09.092126 | hotstack -> 38.102.83.30 | changed
2025-12-03 15:21:09.106650 | 
2025-12-03 15:21:09.106841 | TASK [ocp_agent_installer : Run tasks/pxe_assets.yml]
2025-12-03 15:21:09.155017 | hotstack | ok
2025-12-03 15:21:09.186175 | hotstack | included: /var/lib/zuul/builds/f689c7c4fae14d9e9a8e4ac1bc340a49/trusted/project_3/github.com/openstack-k8s-operators/hotstack/roles/ocp_agent_installer/tasks/pxe_assets.yml
2025-12-03 15:21:09.223924 | 
2025-12-03 15:21:09.224018 | TASK [ocp_agent_installer : Create the boot-artifacts directory]
2025-12-03 15:21:09.667786 | hotstack -> 38.102.83.30 | changed
2025-12-03 15:21:09.684623 | 
2025-12-03 15:21:09.684798 | TASK [ocp_agent_installer : Create PXE assets]
2025-12-03 15:21:11.021002 | controller-0 | level=warning msg=Compute[0].Hyperthreading: Disabled is ignored
2025-12-03 15:21:11.880537 | controller-0 | level=warning msg=ControlPlane.Hyperthreading: Disabled is ignored
2025-12-03 15:21:11.880582 | controller-0 | level=info msg=Configuration has 1 master replicas and 0 worker replicas
2025-12-03 15:21:11.880601 | controller-0 | level=info msg=The rendezvous host IP (node0 IP) is 192.168.32.10
2025-12-03 15:21:13.832038 | controller-0 | level=info msg=Extracting base ISO from release payload
2025-12-03 15:21:50.179305 | controller-0 | level=info msg=Base ISO obtained from release and cached at [/home/zuul/.cache/agent/image_cache/coreos-x86_64.iso]
2025-12-03 15:22:47.854231 | controller-0 | level=info msg=Consuming Extra Manifests from target directory
2025-12-03 15:22:47.854608 | controller-0 | level=info msg=Consuming Agent Config from target directory
2025-12-03 15:22:47.854751 | controller-0 | level=info msg=Consuming Install Config from target directory
2025-12-03 15:22:48.081968 | controller-0 | level=info msg=Created iPXE script agent.x86_64.ipxe in boot-artifacts directory
2025-12-03 15:22:48.082084 | controller-0 | level=info msg=PXE boot artifacts created in: boot-artifacts
2025-12-03 15:22:48.082377 | controller-0 | level=info msg=Kernel parameters for PXE boot: rw ignition.firstboot ignition.platform.id=metal
2025-12-03 15:22:48.564260 | hotstack -> 38.102.83.30 | changed
2025-12-03 15:22:48.570023 | 
2025-12-03 15:22:48.570088 | TASK [ocp_agent_installer : Set serial console in ipxe]
2025-12-03 15:22:49.328849 | hotstack -> 38.102.83.30 | changed: line replaced
2025-12-03 15:22:49.357837 | 
2025-12-03 15:22:49.358219 | TASK [ocp_agent_installer : Disable net.ifnames]
2025-12-03 15:22:49.939313 | hotstack -> 38.102.83.30 | changed: line replaced
2025-12-03 15:22:49.950260 | 
2025-12-03 15:22:49.950397 | LOOP [ocp_agent_installer : Copy boot-artifacts to the web server - (ocp_agent_installer_boot_artifacts_dir)]
2025-12-03 15:22:50.972985 | hotstack -> 38.102.83.30 | changed: "agent.x86_64-vmlinuz"
2025-12-03 15:22:50.973273 | 
2025-12-03 15:22:52.516941 | hotstack -> 38.102.83.30 | changed: "agent.x86_64-initrd.img"
2025-12-03 15:23:06.693239 | hotstack -> 38.102.83.30 | changed: "agent.x86_64-rootfs.img"
2025-12-03 15:23:08.379251 | hotstack -> 38.102.83.30 | changed: "agent.x86_64.ipxe"
2025-12-03 15:23:08.397880 | 
2025-12-03 15:23:08.398065 | TASK [ocp_agent_installer : Run tasks/iso_assets.yml]
2025-12-03 15:23:08.436180 | hotstack | skipping: Conditional result was False
2025-12-03 15:23:08.455470 | 
2025-12-03 15:23:08.455782 | TASK [ocp_agent_installer : Copy auth/kubeconfig to ~/.kube/config]
2025-12-03 15:23:09.036033 | hotstack -> 38.102.83.30 | changed
2025-12-03 15:23:09.045562 | 
2025-12-03 15:23:09.045689 | TASK [ocp_agent_installer : Wait for bootstrap-complete]
2025-12-03 15:23:46.586351 | controller-0 | level=info msg=Waiting for cluster install to initialize. Sleeping for 30 seconds
2025-12-03 15:24:18.589944 | controller-0 | level=info msg=Waiting for cluster install to initialize. Sleeping for 30 seconds
2025-12-03 15:24:50.596279 | controller-0 | level=info msg=Waiting for cluster install to initialize. Sleeping for 30 seconds
2025-12-03 15:25:22.599802 | controller-0 | level=info msg=Cannot access Rendezvous Host. There may be a network configuration problem, check console for additional info
2025-12-03 15:25:54.604570 | controller-0 | level=info msg=Cannot access Rendezvous Host. There may be a network configuration problem, check console for additional info
2025-12-03 15:26:26.616428 | controller-0 | level=info msg=Unable to retrieve cluster metadata from Agent Rest API: no clusterID known for the cluster
2025-12-03 15:26:28.643897 | controller-0 | level=info msg=Cluster is not ready for install. Check validations
2025-12-03 15:26:28.647647 | controller-0 | level=info msg=Successfully registered cluster
2025-12-03 15:26:30.667734 | controller-0 | level=info msg=Updated image information (Image type is "full-iso", SSH public key is set)
2025-12-03 15:26:34.696566 | controller-0 | level=warning msg=Cluster validation: The cluster has hosts that are not ready to install.
2025-12-03 15:26:34.700804 | controller-0 | level=info msg=Host 68855026-7591-4ecb-9a3c-90222f0ae182: Successfully registered
2025-12-03 15:26:40.747135 | controller-0 | level=warning msg=Host master-0 validation: Host couldn't synchronize with any NTP server
2025-12-03 15:26:40.751106 | controller-0 | level=warning msg=Host master-0: updated status from discovering to insufficient (Host cannot be installed due to following failing validation(s): Host couldn't synchronize with any NTP server)
2025-12-03 15:27:37.258055 | controller-0 | level=info msg=Host master-0 validation: Host NTP is synced
2025-12-03 15:27:37.262262 | controller-0 | level=info msg=Host master-0: updated status from insufficient to known (Host is ready to be installed)
2025-12-03 15:27:45.331995 | controller-0 | level=info msg=Preparing cluster for installation
2025-12-03 15:27:45.332460 | controller-0 | level=info msg=Cluster validation: All hosts in the cluster are ready to install.
2025-12-03 15:27:53.405639 | controller-0 | level=info msg=Host master-0: updated status from known to preparing-for-installation (Host finished successfully to prepare for installation)
2025-12-03 15:28:49.888601 | controller-0 | level=info msg=Host master-0: New image status quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2f048f24f6d9c9afe07ab8d64c6aa12a1c3f2be718df791814ee844bafbca660. result: success. time: 1.81 seconds; size: 448.45 Megabytes; download rate: 260.13 MBps
2025-12-03 15:45:04.213298 | controller-0 | level=info msg=Host master-0: updated status from preparing-for-installation to preparing-failed (The host has encountered a preparation timeout, the following conditions failed: the installation disk speed check did not complete within the timeout.)
2025-12-03 15:45:14.298098 | controller-0 | level=info msg=Cluster is ready for install
2025-12-03 15:45:14.298333 | controller-0 | level=info msg=failed to prepare cluster installation, retrying
2025-12-03 15:45:16.309217 | controller-0 | level=info msg=Cluster is ready for install
2025-12-03 15:45:18.323083 | controller-0 | level=info msg=Cluster is ready for install
2025-12-03 15:45:20.333580 | controller-0 | level=warning msg=Cluster validation: The cluster has hosts that are not ready to install.
2025-12-03 15:45:24.365283 | controller-0 | level=info msg=Cluster is ready for install
2025-12-03 15:45:26.382493 | controller-0 | level=info msg=Cluster is ready for install
2025-12-03 15:45:28.393914 | controller-0 | level=info msg=Cluster is ready for install
2025-12-03 15:45:30.411281 | controller-0 | level=info msg=Host master-0: updated status from preparing-failed to known (Host is ready to be installed)
2025-12-03 15:45:34.440956 | controller-0 | level=info msg=Cluster validation: All hosts in the cluster are ready to install.
2025-12-03 15:45:36.464330 | controller-0 | level=info msg=Host master-0: updated status from known to preparing-for-installation (Host finished successfully to prepare for installation)
2025-12-03 16:05:05.103173 | controller-0 | level=info msg=Host master-0: updated status from preparing-for-installation to preparing-failed (The host has encountered a preparation timeout, the following conditions failed: the installation disk speed check did not complete within the timeout.)
2025-12-03 16:05:15.180091 | controller-0 | level=warning msg=Cluster validation: The cluster has hosts that are not ready to install.
2025-12-03 16:06:13.697770 | controller-0 | level=info msg=Cluster is ready for install
2025-12-03 16:06:23.779147 | controller-0 | level=info msg=Cluster is ready for install
2025-12-03 16:06:33.858769 | controller-0 | level=info msg=Cluster is ready for install
2025-12-03 16:06:35.872216 | controller-0 | level=info msg=Cluster is ready for install
2025-12-03 16:06:43.943571 | controller-0 | level=info msg=Cluster is ready for install
2025-12-03 16:06:45.958500 | controller-0 | level=info msg=Cluster is ready for install
2025-12-03 16:06:54.021688 | controller-0 | level=info msg=Cluster is ready for install
2025-12-03 16:06:56.034820 | controller-0 | level=info msg=Cluster is ready for install
2025-12-03 16:06:58.052499 | controller-0 | level=info msg=Host master-0: updated status from preparing-failed to known (Host is ready to be installed)
2025-12-03 16:07:04.097981 | controller-0 | level=info msg=Cluster validation: All hosts in the cluster are ready to install.
2025-12-03 16:07:04.102334 | controller-0 | level=info msg=Host master-0: updated status from known to preparing-for-installation (Host finished successfully to prepare for installation)
2025-12-03 16:23:09.608947 | controller-0 | level=error msg=Attempted to gather ClusterOperator status after wait failure: Listing ClusterOperator objects: Get "https://api.sno.openstack.lab:6443/apis/config.openshift.io/v1/clusteroperators": dial tcp 192.168.32.10:6443: connect: connection refused
2025-12-03 16:23:09.609110 | controller-0 | level=info msg=Use the following commands to gather logs from the cluster
2025-12-03 16:23:09.609172 | controller-0 | level=info msg=openshift-install gather bootstrap --help
2025-12-03 16:23:09.609242 | controller-0 | level=error msg=Bootstrap failed to complete: : bootstrap process timed out: context deadline exceeded
2025-12-03 16:23:10.000889 | hotstack -> 38.102.83.30 | ERROR
2025-12-03 16:23:10.001188 | hotstack -> 38.102.83.30 | {
2025-12-03 16:23:10.001275 | hotstack -> 38.102.83.30 |   "delta": "1:00:00.223100",
2025-12-03 16:23:10.001319 | hotstack -> 38.102.83.30 |   "end": "2025-12-03 16:23:09.621619",
2025-12-03 16:23:10.001356 | hotstack -> 38.102.83.30 |   "msg": "non-zero return code",
2025-12-03 16:23:10.001391 | hotstack -> 38.102.83.30 |   "rc": 5,
2025-12-03 16:23:10.001425 | hotstack -> 38.102.83.30 |   "start": "2025-12-03 15:23:09.398519"
2025-12-03 16:23:10.001459 | hotstack -> 38.102.83.30 | }
failure
2025-12-03 16:23:10.005498 | 
2025-12-03 16:23:10.005752 | PLAY RECAP
2025-12-03 16:23:10.005835 | hotstack | ok: 76 changed: 50 unreachable: 0 failed: 1 skipped: 11 rescued: 0 ignored: 0
2025-12-03 16:23:10.005878 | 
2025-12-03 16:23:10.158266 | PRE-RUN END RESULT_NORMAL: [trusted : review.rdoproject.org/config/playbooks/hotstack/pre_stage2.yaml@master]
2025-12-03 16:23:10.165446 | POST-RUN START: [trusted : review.rdoproject.org/config/playbooks/hotstack/post.yaml@master]
2025-12-03 16:23:10.767026 | 
2025-12-03 16:23:10.767174 | PLAY [Collect hotstack logs and clean up environment]
2025-12-03 16:23:10.790820 | 
2025-12-03 16:23:10.790914 | TASK [Fetch stack outputs from remote to zuul executor]
2025-12-03 16:23:11.458591 | hotstack | Output suppressed because no_log was given
2025-12-03 16:23:11.471411 | 
2025-12-03 16:23:11.471592 | TASK [Load stack output vars with template resolution]
2025-12-03 16:23:11.520734 | hotstack | Output suppressed because no_log was given
2025-12-03 16:23:11.532920 | 
2025-12-03 16:23:11.533066 | TASK [Add Hotstack controller-0 to ansible inventory]
2025-12-03 16:23:11.586993 | hotstack | Output suppressed because no_log was given
2025-12-03 16:23:11.603705 | 
2025-12-03 16:23:11.603893 | TASK [Wait for Hotstack controller-0 to be ready]
2025-12-03 16:23:17.511813 | hotstack -> 38.102.83.30 | Output suppressed because no_log was given
2025-12-03 16:23:17.524421 | 
2025-12-03 16:23:17.524571 | TASK [Collect hotstack logs]
2025-12-03 16:23:17.554000 | hotstack | Output suppressed because no_log was given
2025-12-03 16:23:17.610992 | 
2025-12-03 16:23:17.611137 | TASK [hotlogs : Initialize failure tracking]
2025-12-03 16:23:17.646452 | hotstack | ok
2025-12-03 16:23:17.656583 | 
2025-12-03 16:23:17.656736 | TASK [hotlogs : Ensure logs directory]
2025-12-03 16:23:18.066720 | hotstack | changed
2025-12-03 16:23:18.083460 | 
2025-12-03 16:23:18.083642 | LOOP [hotlogs : Create hotlogs sub dirs]
2025-12-03 16:23:18.325086 | hotstack | changed:
2025-12-03 16:23:18.325538 | hotstack | {
2025-12-03 16:23:18.325602 | hotstack |   "dest_dir": "ocp_cluster",
2025-12-03 16:23:18.325646 | hotstack |   "src": "/home/zuul/ocp-cluster/.openshift_install.log"
2025-12-03 16:23:18.325751 | hotstack | }
2025-12-03 16:23:18.552198 | hotstack | ok:
2025-12-03 16:23:18.552362 | hotstack | {
2025-12-03 16:23:18.552414 | hotstack |   "dest_dir": "ocp_cluster",
2025-12-03 16:23:18.552454 | hotstack |   "src": "/home/zuul/cluster-custom-config/"
2025-12-03 16:23:18.552498 | hotstack | }
2025-12-03 16:23:18.816772 | hotstack | changed:
2025-12-03 16:23:18.816963 | hotstack | {
2025-12-03 16:23:18.817036 | hotstack |   "dest_dir": "data",
2025-12-03 16:23:18.817093 | hotstack |   "src": "/home/zuul/data/"
2025-12-03 16:23:18.817147 | hotstack | }
2025-12-03 16:23:19.054997 | hotstack | changed:
2025-12-03 16:23:19.055502 | hotstack | {
2025-12-03 16:23:19.055597 | hotstack |   "dest_dir": "manifests",
2025-12-03 16:23:19.055655 | hotstack |   "src": "/home/zuul/manifests/"
2025-12-03 16:23:19.055759 | hotstack | }
2025-12-03 16:23:19.306960 | hotstack | ok:
2025-12-03 16:23:19.307118 | hotstack | {
2025-12-03 16:23:19.307168 | hotstack |   "dest_dir": ".",
2025-12-03 16:23:19.307227 | hotstack |   "src": "/home/zuul/must-gather.tar.gz"
2025-12-03 16:23:19.307269 | hotstack | }
2025-12-03 16:23:19.531929 | hotstack | ok:
2025-12-03 16:23:19.532118 | hotstack | {
2025-12-03 16:23:19.532185 | hotstack |   "dest_dir": ".",
2025-12-03 16:23:19.532238 | hotstack |   "src": "/home/zuul/logs/tempest/"
2025-12-03 16:23:19.532311 | hotstack | }
2025-12-03 16:23:19.557010 | 
2025-12-03 16:23:19.557205 | TASK [hotlogs : Run must-gather operations]
2025-12-03 16:23:20.791050 | hotstack -> 38.102.83.30 | ERROR
2025-12-03 16:23:20.791491 | hotstack -> 38.102.83.30 | {
2025-12-03 16:23:20.791555 | hotstack -> 38.102.83.30 |   "msg": "Must-gather failed: error: unable to resolve image stream 'openshift/must-gather': Get \"https://api.sno.openstack.lab:6443/apis/image.openshift.io/v1/namespaces/openshift/imagestreams/must-gather\": dial tcp 192.168.32.10:6443: connect: connection refused\n (Command: oc adm must-gather --image-stream=openshift/must-gather --image=quay.io/openstack-k8s-operators/openstack-must-gather --dest-dir=/home/zuul/must-gather --timeout=10m --host-network=false -- ADDITIONAL_NAMESPACES=sushy-emulator OPENSTACK_DATABASES=ALL SOS_EDPM=all SOS_DECOMPRESS=0 gather)",
2025-12-03 16:23:20.791605 | hotstack -> 38.102.83.30 |   "rc": 1
2025-12-03 16:23:20.791645 | hotstack -> 38.102.83.30 | }
2025-12-03 16:23:20.847028 | 
2025-12-03 16:23:20.847173 | TASK [hotlogs : Display must-gather failure]
2025-12-03 16:23:20.921024 | hotstack | ok:
2025-12-03 16:23:20.921345 | hotstack | Must-gather operations failed: Must-gather failed: error: unable to resolve image stream 'openshift/must-gather': Get "https://api.sno.openstack.lab:6443/apis/image.openshift.io/v1/namespaces/openshift/imagestreams/must-gather": dial tcp 192.168.32.10:6443: connect: connection refused
2025-12-03 16:23:20.921386 | hotstack |  (Command: oc adm must-gather --image-stream=openshift/must-gather --image=quay.io/openstack-k8s-operators/openstack-must-gather --dest-dir=/home/zuul/must-gather --timeout=10m --host-network=false -- ADDITIONAL_NAMESPACES=sushy-emulator OPENSTACK_DATABASES=ALL SOS_EDPM=all SOS_DECOMPRESS=0 gather)
2025-12-03 16:23:20.928591 | 
2025-12-03 16:23:20.928699 | TASK [hotlogs : Track must-gather failure]
2025-12-03 16:23:20.982358 | hotstack | ok
2025-12-03 16:23:20.990623 | 
2025-12-03 16:23:20.990727 | LOOP [hotlogs : Collect tempest logs from test-operator PVCs]
2025-12-03 16:23:21.927046 | hotstack -> 38.102.83.30 | ok: non-zero return code
2025-12-03 16:23:21.927474 | hotstack -> 38.102.83.30 | changed: All items complete
2025-12-03 16:23:21.927536 | 
2025-12-03 16:23:21.947431 | 
2025-12-03 16:23:21.947624 | LOOP [hotlogs : Display tempest log collection results]
2025-12-03 16:23:22.026924 | hotstack | ok: Namespace openstack: ['==========================================', 'Tempest Log Collection', '==========================================', 'Namespace: openstack', 'Logs directory: /home/zuul/logs/tempest', '==========================================', '', 'Searching for tempest log PVCs...', 'E1203 16:23:21.873326   14556 memcache.go:265] "Unhandled Error" err="couldn\'t get current server API group list: Get \\"https://api.sno.openstack.lab:6443/api?timeout=32s\\": dial tcp 192.168.32.10:6443: connect: connection refused"', 'E1203 16:23:21.876878   14556 memcache.go:265] "Unhandled Error" err="couldn\'t get current server API group list: Get \\"https://api.sno.openstack.lab:6443/api?timeout=32s\\": dial tcp 192.168.32.10:6443: connect: connection refused"', 'E1203 16:23:21.878645   14556 memcache.go:265] "Unhandled Error" err="couldn\'t get current server API group list: Get \\"https://api.sno.openstack.lab:6443/api?timeout=32s\\": dial tcp 192.168.32.10:6443: connect: connection refused"', 'E1203 16:23:21.880438   14556 memcache.go:265] "Unhandled Error" err="couldn\'t get current server API group list: Get \\"https://api.sno.openstack.lab:6443/api?timeout=32s\\": dial tcp 192.168.32.10:6443: connect: connection refused"', 'E1203 16:23:21.882228   14556 memcache.go:265] "Unhandled Error" err="couldn\'t get current server API group list: Get \\"https://api.sno.openstack.lab:6443/api?timeout=32s\\": dial tcp 192.168.32.10:6443: connect: connection refused"', 'The connection to the server api.sno.openstack.lab:6443 was refused - did you specify the right host or port?']
2025-12-03 16:23:22.041184 | 
2025-12-03 16:23:22.041365 | LOOP [hotlogs : Track tempest log collection failures]
2025-12-03 16:23:22.092066 | hotstack | ok:
2025-12-03 16:23:22.092383 | hotstack | {
2025-12-03 16:23:22.092605 | hotstack |   "ansible_loop_var": "item",
2025-12-03 16:23:22.092749 | hotstack |   "changed": false,
2025-12-03 16:23:22.092806 | hotstack |   "cmd": [
2025-12-03 16:23:22.092852 | hotstack |     "hotstack-collect-tempest-logs",
2025-12-03 16:23:22.092896 | hotstack |     "--namespace",
2025-12-03 16:23:22.092938 | hotstack |     "openstack"
2025-12-03 16:23:22.092980 | hotstack |   ],
2025-12-03 16:23:22.093022 | hotstack |   "delta": "0:00:00.239780",
2025-12-03 16:23:22.093064 | hotstack |   "end": "2025-12-03 16:23:21.892041",
2025-12-03 16:23:22.093106 | hotstack |   "failed": false,
2025-12-03 16:23:22.093150 | hotstack |   "failed_when_result": false,
2025-12-03 16:23:22.093193 | hotstack |   "invocation": {
2025-12-03 16:23:22.093234 | hotstack |     "module_args": {
2025-12-03 16:23:22.093275 | hotstack |       "_raw_params": "hotstack-collect-tempest-logs --namespace openstack",
2025-12-03 16:23:22.093317 | hotstack |       "_uses_shell": false,
2025-12-03 16:23:22.093358 | hotstack |       "argv": null,
2025-12-03 16:23:22.093399 | hotstack |       "chdir": null,
2025-12-03 16:23:22.093440 | hotstack |       "creates": null,
2025-12-03 16:23:22.093481 | hotstack |       "executable": null,
2025-12-03 16:23:22.093523 | hotstack |       "removes": null,
2025-12-03 16:23:22.093563 | hotstack |       "stdin": null,
2025-12-03 16:23:22.093793 | hotstack |       "stdin_add_newline": true,
2025-12-03 16:23:22.093864 | hotstack |       "strip_empty_ends": true,
2025-12-03 16:23:22.093936 | hotstack |       "warn": false,
2025-12-03 16:23:22.093985 | hotstack |       "zuul_ansible_split_streams": false,
2025-12-03 16:23:22.094030 | hotstack |       "zuul_log_id": "in-loop-ignore"
2025-12-03 16:23:22.094073 | hotstack |     }
2025-12-03 16:23:22.094114 | hotstack |   },
2025-12-03 16:23:22.094156 | hotstack |   "item": "openstack",
2025-12-03 16:23:22.094197 | hotstack |   "msg": "non-zero return code",
2025-12-03 16:23:22.094238 | hotstack |   "rc": 1,
2025-12-03 16:23:22.094278 | hotstack |   "start": "2025-12-03 16:23:21.652261",
2025-12-03 16:23:22.094319 | hotstack |   "stderr": "",
2025-12-03 16:23:22.094359 | hotstack |   "stderr_lines": [],
2025-12-03 16:23:22.094400 | hotstack |   "stdout": "==========================================\nTempest Log Collection\n==========================================\nNamespace: openstack\nLogs directory: /home/zuul/logs/tempest\n==========================================\n\nSearching for tempest log PVCs...\nE1203 16:23:21.873326   14556 memcache.go:265] \"Unhandled Error\" err=\"couldn't get current server API group list: Get \\\"https://api.sno.openstack.lab:6443/api?timeout=32s\\\": dial tcp 192.168.32.10:6443: connect: connection refused\"\nE1203 16:23:21.876878   14556 memcache.go:265] \"Unhandled Error\" err=\"couldn't get current server API group list: Get \\\"https://api.sno.openstack.lab:6443/api?timeout=32s\\\": dial tcp 192.168.32.10:6443: connect: connection refused\"\nE1203 16:23:21.878645   14556 memcache.go:265] \"Unhandled Error\" err=\"couldn't get current server API group list: Get \\\"https://api.sno.openstack.lab:6443/api?timeout=32s\\\": dial tcp 192.168.32.10:6443: connect: connection refused\"\nE1203 16:23:21.880438   14556 memcache.go:265] \"Unhandled Error\" err=\"couldn't get current server API group list: Get \\\"https://api.sno.openstack.lab:6443/api?timeout=32s\\\": dial tcp 192.168.32.10:6443: connect: connection refused\"\nE1203 16:23:21.882228   14556 memcache.go:265] \"Unhandled Error\" err=\"couldn't get current server API group list: Get \\\"https://api.sno.openstack.lab:6443/api?timeout=32s\\\": dial tcp 192.168.32.10:6443: connect: connection refused\"\nThe connection to the server api.sno.openstack.lab:6443 was refused - did you specify the right host or port?",
2025-12-03 16:23:22.094446 | hotstack |   "stdout_lines": [
2025-12-03 16:23:22.094488 | hotstack |     "==========================================",
2025-12-03 16:23:22.094950 | hotstack |     "Tempest Log Collection",
2025-12-03 16:23:22.095041 | hotstack |     "==========================================",
2025-12-03 16:23:22.095086 | hotstack |     "Namespace: openstack",
2025-12-03 16:23:22.095123 | hotstack |     "Logs directory: /home/zuul/logs/tempest",
2025-12-03 16:23:22.095158 | hotstack |     "==========================================",
2025-12-03 16:23:22.095192 | hotstack |     "",
2025-12-03 16:23:22.095228 | hotstack |     "Searching for tempest log PVCs...",
2025-12-03 16:23:22.095264 | hotstack |     "E1203 16:23:21.873326   14556 memcache.go:265] \"Unhandled Error\" err=\"couldn't get current server API group list: Get \\\"https://api.sno.openstack.lab:6443/api?timeout=32s\\\": dial tcp 192.168.32.10:6443: connect: connection refused\"",
2025-12-03 16:23:22.095301 | hotstack |     "E1203 16:23:21.876878   14556 memcache.go:265] \"Unhandled Error\" err=\"couldn't get current server API group list: Get \\\"https://api.sno.openstack.lab:6443/api?timeout=32s\\\": dial tcp 192.168.32.10:6443: connect: connection refused\"",
2025-12-03 16:23:22.095342 | hotstack |     "E1203 16:23:21.878645   14556 memcache.go:265] \"Unhandled Error\" err=\"couldn't get current server API group list: Get \\\"https://api.sno.openstack.lab:6443/api?timeout=32s\\\": dial tcp 192.168.32.10:6443: connect: connection refused\"",
2025-12-03 16:23:22.095378 | hotstack |     "E1203 16:23:21.880438   14556 memcache.go:265] \"Unhandled Error\" err=\"couldn't get current server API group list: Get \\\"https://api.sno.openstack.lab:6443/api?timeout=32s\\\": dial tcp 192.168.32.10:6443: connect: connection refused\"",
2025-12-03 16:23:22.095412 | hotstack |     "E1203 16:23:21.882228   14556 memcache.go:265] \"Unhandled Error\" err=\"couldn't get current server API group list: Get \\\"https://api.sno.openstack.lab:6443/api?timeout=32s\\\": dial tcp 192.168.32.10:6443: connect: connection refused\"",
2025-12-03 16:23:22.095446 | hotstack |     "The connection to the server api.sno.openstack.lab:6443 was refused - did you specify the right host or port?"
2025-12-03 16:23:22.095482 | hotstack |   ],
2025-12-03 16:23:22.095517 | hotstack |   "zuul_log_id": "in-loop-ignore"
2025-12-03 16:23:22.095552 | hotstack | }
2025-12-03 16:23:22.095616 | hotstack | ok: All items complete
2025-12-03 16:23:22.095658 | 
2025-12-03 16:23:22.119451 | 
2025-12-03 16:23:22.119632 | LOOP [hotlogs : Collect hotstack data]
2025-12-03 16:23:24.354813 | hotstack -> 38.102.83.220 | changed: >f+++++++++ .openshift_install.log
2025-12-03 16:23:24.935965 | hotstack -> 38.102.83.220 | changed:
2025-12-03 16:23:24.936135 | hotstack -> 38.102.83.220 | .d..t...... ./
2025-12-03 16:23:24.936189 | hotstack -> 38.102.83.220 | cd+++++++++ butane/
2025-12-03 16:23:24.936230 | hotstack -> 38.102.83.220 | >f+++++++++ butane/90-master-disable-netifnames.bu
2025-12-03 16:23:24.936270 | hotstack -> 38.102.83.220 | >f+++++++++ butane/90-master-enable-iscsi.bu
2025-12-03 16:23:24.936409 | hotstack -> 38.102.83.220 | >f+++++++++ butane/91-master-enable-multipath.bu
2025-12-03 16:23:24.936458 | hotstack -> 38.102.83.220 | >f+++++++++ butane/92-master-lv-cinder-volumes.bu
2025-12-03 16:23:24.936495 | hotstack -> 38.102.83.220 | cd+++++++++ config-assets/
2025-12-03 16:23:24.936531 | hotstack -> 38.102.83.220 | >f+++++++++ config-assets/95-etcd_config.yaml
2025-12-03 16:23:24.936590 | hotstack -> 38.102.83.220 | >f+++++++++ config-assets/ovn_k8s_config.yaml
2025-12-03 16:23:24.936627 | hotstack -> 38.102.83.220 | cd+++++++++ machine-configs/
2025-12-03 16:23:24.936661 | hotstack -> 38.102.83.220 | >f+++++++++ machine-configs/90-master-disable-netifnames.yaml
2025-12-03 16:23:24.936745 | hotstack -> 38.102.83.220 | >f+++++++++ machine-configs/90-master-enable-iscsi.yaml
2025-12-03 16:23:24.936781 | hotstack -> 38.102.83.220 | >f+++++++++ machine-configs/91-master-enable-multipath.yaml
2025-12-03 16:23:24.936816 | hotstack -> 38.102.83.220 | >f+++++++++ machine-configs/92-master-lv-cinder-volumes.yaml
2025-12-03 16:23:25.557453 | hotstack -> 38.102.83.220 | changed:
2025-12-03 16:23:25.557615 | hotstack -> 38.102.83.220 | .d..t...... ./
2025-12-03 16:23:25.557716 | hotstack -> 38.102.83.220 | >f+++++++++ ironic_nodes.yaml
2025-12-03 16:23:26.163882 | hotstack -> 38.102.83.220 | ERROR: Item: {'src': '/home/zuul/manifests/', 'dest_dir': 'manifests'}
2025-12-03 16:23:26.164175 | hotstack -> 38.102.83.220 | {
2025-12-03 16:23:26.164240 | hotstack -> 38.102.83.220 |   "ansible_loop_var": "item",
2025-12-03 16:23:26.164281 | hotstack -> 38.102.83.220 |   "item": {
2025-12-03 16:23:26.164318 | hotstack -> 38.102.83.220 |     "dest_dir": "manifests",
2025-12-03 16:23:26.164353 | hotstack -> 38.102.83.220 |     "src": "/home/zuul/manifests/"
2025-12-03 16:23:26.164387 | hotstack -> 38.102.83.220 |   },
2025-12-03 16:23:26.164422 | hotstack -> 38.102.83.220 |   "msg": "Warning: Permanently added '38.102.83.30' (ED25519) to the list of known hosts.\r\nrsync: [sender] change_dir \"/home/zuul/manifests\" failed: No such file or directory (2)\nrsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1841) [Receiver=3.2.5]\nrsync: [Receiver] write error: Broken pipe (32)\n",
2025-12-03 16:23:26.164462 | hotstack -> 38.102.83.220 |   "rc": 23
2025-12-03 16:23:26.164497 | hotstack -> 38.102.83.220 | }
2025-12-03 16:23:26.837097 | hotstack -> 38.102.83.220 | ERROR: Item: {'src': '/home/zuul/must-gather.tar.gz', 'dest_dir': '.'}
2025-12-03 16:23:26.837317 | hotstack -> 38.102.83.220 | {
2025-12-03 16:23:26.837377 | hotstack -> 38.102.83.220 |   "ansible_loop_var": "item",
2025-12-03 16:23:26.837419 | hotstack -> 38.102.83.220 |   "item": {
2025-12-03 16:23:26.837455 | hotstack -> 38.102.83.220 |     "dest_dir": ".",
2025-12-03 16:23:26.837490 | hotstack -> 38.102.83.220 |     "src": "/home/zuul/must-gather.tar.gz"
2025-12-03 16:23:26.837525 | hotstack -> 38.102.83.220 |   },
2025-12-03 16:23:26.837561 | hotstack -> 38.102.83.220 |   "msg": "Warning: Permanently added '38.102.83.30' (ED25519) to the list of known hosts.\r\nrsync: [sender] link_stat \"/home/zuul/must-gather.tar.gz\" failed: No such file or directory (2)\nrsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1841) [Receiver=3.2.5]\nrsync: [Receiver] write error: Broken pipe (32)\n",
2025-12-03 16:23:26.837606 | hotstack -> 38.102.83.220 |   "rc": 23
2025-12-03 16:23:26.837641 | hotstack -> 38.102.83.220 | }
2025-12-03 16:23:27.322586 | hotstack -> 38.102.83.220 | changed: .d..t...... ./
2025-12-03 16:23:27.342192 | hotstack | ERROR: Ignoring Errors
2025-12-03 16:23:27.359163 | 
2025-12-03 16:23:27.359329 | LOOP [hotlogs : Check hotstack data collection failures]
2025-12-03 16:23:27.433039 | hotstack | skipping: Conditional result was False
2025-12-03 16:23:27.440641 | hotstack | skipping: Conditional result was False
2025-12-03 16:23:27.448003 | hotstack | skipping: Conditional result was False
2025-12-03 16:23:27.471093 | hotstack | ERROR
2025-12-03 16:23:27.471265 | hotstack | {
2025-12-03 16:23:27.471323 | hotstack |   "msg": "Unexpected templating type error occurred on ({{ hotlogs_failures + ['Failed to collect: ' + item.item] }}): can only concatenate str (not \"dict\") to str. can only concatenate str (not \"dict\") to str"
2025-12-03 16:23:27.471363 | hotstack | }
2025-12-03 16:23:27.482293 | 
2025-12-03 16:23:27.482420 | TASK [Print message that log collection failed]
2025-12-03 16:23:27.521363 | hotstack | ok: WARNING: Log collection failed and may be incomplete. Continue with cleanup ...
2025-12-03 16:23:27.534627 | 
2025-12-03 16:23:27.534821 | TASK [Set _auth_type fact]
2025-12-03 16:23:27.586830 | hotstack | Output suppressed because no_log was given
2025-12-03 16:23:27.592199 | 
2025-12-03 16:23:27.592266 | TASK [Write clouds.yaml]
2025-12-03 16:23:28.556086 | hotstack | Output suppressed because no_log was given
2025-12-03 16:23:28.567123 | 
2025-12-03 16:23:28.567254 | TASK [Delete application credentials]
2025-12-03 16:23:30.621579 | hotstack | changed
2025-12-03 16:23:30.634838 | 
2025-12-03 16:23:30.634994 | TASK [Print message that application credential failed to delete]
2025-12-03 16:23:30.662251 | hotstack | skipping: Conditional result was False
2025-12-03 16:23:30.675437 | 
2025-12-03 16:23:30.675577 | TASK [Print message that application credential deleted successfully]
2025-12-03 16:23:30.722648 | hotstack | ok: Application credential deleted successfully.
2025-12-03 16:23:30.733285 | 
2025-12-03 16:23:30.733416 | TASK [Write clouds.yaml with application credentials]
2025-12-03 16:23:30.771580 | 
2025-12-03 16:23:30.771932 | TASK [Delete the heat stack]
2025-12-03 11:23:53.007891 | hotstack | 2025-12-03 16:23:33Z [hs-f689c7c4fae14d9e9a8e4ac1bc340a49]: DELETE_IN_PROGRESS  Stack DELETE started
2025-12-03 11:23:53.007980 | hotstack | 2025-12-03 16:23:33Z [hs-f689c7c4fae14d9e9a8e4ac1bc340a49.ironic1]: DELETE_IN_PROGRESS  state changed
2025-12-03 11:23:53.007992 | hotstack | 2025-12-03 16:23:34Z [hs-f689c7c4fae14d9e9a8e4ac1bc340a49.controller-floating-ip]: DELETE_IN_PROGRESS  state changed
2025-12-03 11:23:53.008000 | hotstack | 2025-12-03 16:23:35Z [hs-f689c7c4fae14d9e9a8e4ac1bc340a49.ironic-net-router-interface]: DELETE_IN_PROGRESS  state changed
2025-12-03 11:23:53.008009 | hotstack | 2025-12-03 16:23:36Z [hs-f689c7c4fae14d9e9a8e4ac1bc340a49.controller-floating-ip]: DELETE_COMPLETE  state changed
2025-12-03 11:23:53.008017 | hotstack | 2025-12-03 16:23:36Z [hs-f689c7c4fae14d9e9a8e4ac1bc340a49.machine-net-router-interface]: DELETE_IN_PROGRESS  state changed
2025-12-03 11:23:53.008025 | hotstack | 2025-12-03 16:23:36Z [hs-f689c7c4fae14d9e9a8e4ac1bc340a49.ironic0]: DELETE_IN_PROGRESS  state changed
2025-12-03 11:23:53.008033 | hotstack | 2025-12-03 16:23:37Z [hs-f689c7c4fae14d9e9a8e4ac1bc340a49.controller]: DELETE_IN_PROGRESS  state changed
2025-12-03 11:23:53.008041 | hotstack | 2025-12-03 16:23:37Z [hs-f689c7c4fae14d9e9a8e4ac1bc340a49.ironic1]: DELETE_COMPLETE  state changed
2025-12-03 11:23:53.008049 | hotstack | 2025-12-03 16:23:37Z [hs-f689c7c4fae14d9e9a8e4ac1bc340a49.ironic1-port]: DELETE_IN_PROGRESS  state changed
2025-12-03 11:23:53.008057 | hotstack | 2025-12-03 16:23:37Z [hs-f689c7c4fae14d9e9a8e4ac1bc340a49.ironic-net-router-interface]: DELETE_COMPLETE  state changed
2025-12-03 11:23:53.008066 | hotstack | 2025-12-03 16:23:38Z [hs-f689c7c4fae14d9e9a8e4ac1bc340a49.master0]: DELETE_IN_PROGRESS  state changed
2025-12-03 11:23:53.008073 | hotstack | 2025-12-03 16:23:38Z [hs-f689c7c4fae14d9e9a8e4ac1bc340a49.ironic1-port]: DELETE_COMPLETE  state changed
2025-12-03 11:23:53.008081 | hotstack | 2025-12-03 16:23:38Z [hs-f689c7c4fae14d9e9a8e4ac1bc340a49.machine-net-router-interface]: DELETE_COMPLETE  state changed
2025-12-03 11:23:53.008089 | hotstack | 2025-12-03 16:23:39Z [hs-f689c7c4fae14d9e9a8e4ac1bc340a49.ctlplane-net-router-interface]: DELETE_IN_PROGRESS  state changed
2025-12-03 11:23:53.008097 | hotstack | 2025-12-03 16:23:41Z [hs-f689c7c4fae14d9e9a8e4ac1bc340a49.ctlplane-net-router-interface]: DELETE_COMPLETE  state changed
2025-12-03 11:23:53.008113 | hotstack | 2025-12-03 16:23:41Z [hs-f689c7c4fae14d9e9a8e4ac1bc340a49.router]: DELETE_IN_PROGRESS  state changed
2025-12-03 11:23:53.008121 | hotstack | 2025-12-03 16:23:43Z [hs-f689c7c4fae14d9e9a8e4ac1bc340a49.router]: DELETE_COMPLETE  state changed
2025-12-03 11:23:53.008128 | hotstack | 2025-12-03 16:23:44Z [hs-f689c7c4fae14d9e9a8e4ac1bc340a49.ironic0]: DELETE_COMPLETE  state changed
2025-12-03 11:23:53.008136 | hotstack | 2025-12-03 16:23:44Z [hs-f689c7c4fae14d9e9a8e4ac1bc340a49.ironic0-port]: DELETE_IN_PROGRESS  state changed
2025-12-03 11:23:53.008149 | hotstack | 2025-12-03 16:23:44Z [hs-f689c7c4fae14d9e9a8e4ac1bc340a49.ironic0-port]: DELETE_COMPLETE  state changed
2025-12-03 11:23:53.008177 | hotstack | 2025-12-03 16:23:46Z [hs-f689c7c4fae14d9e9a8e4ac1bc340a49.master0]: DELETE_COMPLETE  state changed
2025-12-03 11:23:53.008185 | hotstack | 2025-12-03 16:23:46Z [hs-f689c7c4fae14d9e9a8e4ac1bc340a49.master0-cinder-vol1]: DELETE_IN_PROGRESS  state changed
2025-12-03 11:23:53.008193 | hotstack | 2025-12-03 16:23:46Z [hs-f689c7c4fae14d9e9a8e4ac1bc340a49.master0-cinder-vol0]: DELETE_IN_PROGRESS  state changed
2025-12-03 11:23:53.008200 | hotstack | 2025-12-03 16:23:47Z [hs-f689c7c4fae14d9e9a8e4ac1bc340a49.master0-cinder-vol2]: DELETE_IN_PROGRESS  state changed
2025-12-03 11:23:53.008208 | hotstack | 2025-12-03 16:23:47Z [hs-f689c7c4fae14d9e9a8e4ac1bc340a49.master0-lvms-vol0]: DELETE_IN_PROGRESS  state changed
2025-12-03 11:23:53.008215 | hotstack | 2025-12-03 16:23:47Z [hs-f689c7c4fae14d9e9a8e4ac1bc340a49.master0-trunk0]: DELETE_IN_PROGRESS  state changed
2025-12-03 11:23:53.008223 | hotstack | 2025-12-03 16:23:47Z [hs-f689c7c4fae14d9e9a8e4ac1bc340a49.master0-ironic-port]: DELETE_IN_PROGRESS  state changed
2025-12-03 11:23:53.008231 | hotstack | 2025-12-03 16:23:47Z [hs-f689c7c4fae14d9e9a8e4ac1bc340a49.controller]: DELETE_COMPLETE  state changed
2025-12-03 11:23:53.008238 | hotstack | 2025-12-03 16:23:47Z [hs-f689c7c4fae14d9e9a8e4ac1bc340a49.master0-ironic-port]: DELETE_COMPLETE  state changed
2025-12-03 11:23:53.008254 | hotstack | 2025-12-03 16:23:47Z [hs-f689c7c4fae14d9e9a8e4ac1bc340a49.controller-init]: DELETE_IN_PROGRESS  state changed
2025-12-03 11:23:53.008264 | hotstack | 2025-12-03 16:23:47Z [hs-f689c7c4fae14d9e9a8e4ac1bc340a49.controller-init]: DELETE_COMPLETE  state changed
2025-12-03 11:23:53.008275 | hotstack | 2025-12-03 16:23:47Z [hs-f689c7c4fae14d9e9a8e4ac1bc340a49.ironic-subnet]: DELETE_IN_PROGRESS  state changed
2025-12-03 11:23:53.008284 | hotstack | 2025-12-03 16:23:47Z [hs-f689c7c4fae14d9e9a8e4ac1bc340a49.master0-trunk0]: DELETE_COMPLETE  state changed
2025-12-03 11:23:53.008292 | hotstack | 2025-12-03 16:23:47Z [hs-f689c7c4fae14d9e9a8e4ac1bc340a49.controller-write-files]: DELETE_IN_PROGRESS  state changed
2025-12-03 11:23:53.008300 | hotstack | 2025-12-03 16:23:47Z [hs-f689c7c4fae14d9e9a8e4ac1bc340a49.controller-runcmd]: DELETE_IN_PROGRESS  state changed
2025-12-03 11:23:53.008308 | hotstack | 2025-12-03 16:23:47Z [hs-f689c7c4fae14d9e9a8e4ac1bc340a49.controller_users]: DELETE_IN_PROGRESS  state changed
2025-12-03 11:23:53.008321 | hotstack | 2025-12-03 16:23:47Z [hs-f689c7c4fae14d9e9a8e4ac1bc340a49.master0-tenant-port]: DELETE_IN_PROGRESS  state changed
2025-12-03 11:23:53.008330 | hotstack | 2025-12-03 16:23:47Z [hs-f689c7c4fae14d9e9a8e4ac1bc340a49.controller-write-files]: DELETE_COMPLETE  state changed
2025-12-03 11:23:53.008342 | hotstack | 2025-12-03 16:23:47Z [hs-f689c7c4fae14d9e9a8e4ac1bc340a49.controller-runcmd]: DELETE_COMPLETE  state changed
2025-12-03 11:23:53.008351 | hotstack | 2025-12-03 16:23:47Z [hs-f689c7c4fae14d9e9a8e4ac1bc340a49.master0-storage-port]: DELETE_IN_PROGRESS  state changed
2025-12-03 11:23:53.008359 | hotstack | 2025-12-03 16:23:48Z [hs-f689c7c4fae14d9e9a8e4ac1bc340a49.master0-ctlplane-trunk-parent-port]: DELETE_IN_PROGRESS  state changed
2025-12-03 11:23:53.008366 | hotstack | 2025-12-03 16:23:48Z [hs-f689c7c4fae14d9e9a8e4ac1bc340a49.master0-internal-api-port]: DELETE_IN_PROGRESS  state changed
2025-12-03 11:23:53.008374 | hotstack | 2025-12-03 16:23:48Z [hs-f689c7c4fae14d9e9a8e4ac1bc340a49.controller_users]: DELETE_COMPLETE  state changed
2025-12-03 11:23:53.008383 | hotstack | 2025-12-03 16:23:48Z [hs-f689c7c4fae14d9e9a8e4ac1bc340a49.master0-machine-port]: DELETE_IN_PROGRESS  state changed
2025-12-03 11:23:53.008391 | hotstack | 2025-12-03 16:23:48Z [hs-f689c7c4fae14d9e9a8e4ac1bc340a49.master0-cinder-vol1]: DELETE_COMPLETE  state changed
2025-12-03 11:23:53.008399 | hotstack | 2025-12-03 16:23:48Z [hs-f689c7c4fae14d9e9a8e4ac1bc340a49.ironic-subnet]: DELETE_COMPLETE  state changed
2025-12-03 11:23:53.008407 | hotstack | 2025-12-03 16:23:48Z [hs-f689c7c4fae14d9e9a8e4ac1bc340a49.master0-cinder-vol0]: DELETE_COMPLETE  state changed
2025-12-03 11:23:53.008414 | hotstack | 2025-12-03 16:23:48Z [hs-f689c7c4fae14d9e9a8e4ac1bc340a49.ironic-net]: DELETE_IN_PROGRESS  state changed
2025-12-03 11:23:53.008422 | hotstack | 2025-12-03 16:23:48Z [hs-f689c7c4fae14d9e9a8e4ac1bc340a49.master0-cinder-vol2]: DELETE_COMPLETE  state changed
2025-12-03 11:23:53.008430 | hotstack | 2025-12-03 16:23:48Z [hs-f689c7c4fae14d9e9a8e4ac1bc340a49.master0-lvms-vol0]: DELETE_COMPLETE  state changed
2025-12-03 11:23:53.008438 | hotstack | 2025-12-03 16:23:48Z [hs-f689c7c4fae14d9e9a8e4ac1bc340a49.master0-storage-port]: DELETE_COMPLETE  state changed
2025-12-03 11:23:53.008449 | hotstack | 2025-12-03 16:23:48Z [hs-f689c7c4fae14d9e9a8e4ac1bc340a49.storage-subnet]: DELETE_IN_PROGRESS  state changed
2025-12-03 11:23:53.008457 | hotstack | 2025-12-03 16:23:48Z [hs-f689c7c4fae14d9e9a8e4ac1bc340a49.master0-tenant-port]: DELETE_COMPLETE  state changed
2025-12-03 11:23:53.008464 | hotstack | 2025-12-03 16:23:48Z [hs-f689c7c4fae14d9e9a8e4ac1bc340a49.tenant-subnet]: DELETE_IN_PROGRESS  state changed
2025-12-03 11:23:53.008472 | hotstack | 2025-12-03 16:23:49Z [hs-f689c7c4fae14d9e9a8e4ac1bc340a49.master0-internal-api-port]: DELETE_COMPLETE  state changed
2025-12-03 11:23:53.008480 | hotstack | 2025-12-03 16:23:49Z [hs-f689c7c4fae14d9e9a8e4ac1bc340a49.storage-subnet]: DELETE_COMPLETE  state changed
2025-12-03 11:23:53.008489 | hotstack | 2025-12-03 16:23:49Z [hs-f689c7c4fae14d9e9a8e4ac1bc340a49.master0-ctlplane-trunk-parent-port]: DELETE_COMPLETE  state changed
2025-12-03 11:23:53.008497 | hotstack | 2025-12-03 16:23:49Z [hs-f689c7c4fae14d9e9a8e4ac1bc340a49.internal-api-subnet]: DELETE_IN_PROGRESS  state changed
2025-12-03 11:23:53.008508 | hotstack | 2025-12-03 16:23:49Z [hs-f689c7c4fae14d9e9a8e4ac1bc340a49.storage-net]: DELETE_IN_PROGRESS  state changed
2025-12-03 11:23:53.008516 | hotstack | 2025-12-03 16:23:49Z [hs-f689c7c4fae14d9e9a8e4ac1bc340a49.ctlplane-subnet]: DELETE_IN_PROGRESS  state changed
2025-12-03 11:23:53.008523 | hotstack | 2025-12-03 16:23:49Z [hs-f689c7c4fae14d9e9a8e4ac1bc340a49.tenant-subnet]: DELETE_COMPLETE  state changed
2025-12-03 11:23:53.008531 | hotstack | 2025-12-03 16:23:49Z [hs-f689c7c4fae14d9e9a8e4ac1bc340a49.tenant-net]: DELETE_IN_PROGRESS  state changed
2025-12-03 11:23:53.008538 | hotstack | 2025-12-03 16:23:49Z [hs-f689c7c4fae14d9e9a8e4ac1bc340a49.ironic-net]: DELETE_COMPLETE  state changed
2025-12-03 11:23:53.008551 | hotstack | 2025-12-03 16:23:49Z [hs-f689c7c4fae14d9e9a8e4ac1bc340a49.master0-machine-port]: DELETE_COMPLETE  state changed
2025-12-03 11:23:53.008559 | hotstack | 2025-12-03 16:23:49Z [hs-f689c7c4fae14d9e9a8e4ac1bc340a49.internal-api-subnet]: DELETE_COMPLETE  state changed
2025-12-03 11:23:53.008567 | hotstack | 2025-12-03 16:23:49Z [hs-f689c7c4fae14d9e9a8e4ac1bc340a49.extra-dhcp-opts-value]: DELETE_IN_PROGRESS  state changed
2025-12-03 11:23:53.008575 | hotstack | 2025-12-03 16:23:49Z [hs-f689c7c4fae14d9e9a8e4ac1bc340a49.extra-dhcp-opts-value]: DELETE_COMPLETE  state changed
2025-12-03 11:23:53.008582 | hotstack | 2025-12-03 16:23:49Z [hs-f689c7c4fae14d9e9a8e4ac1bc340a49.internal-api-net]: DELETE_IN_PROGRESS  state changed
2025-12-03 11:23:53.008608 | hotstack | 2025-12-03 16:23:49Z [hs-f689c7c4fae14d9e9a8e4ac1bc340a49.controller-machine-port]: DELETE_IN_PROGRESS  state changed
2025-12-03 11:23:53.008616 | hotstack | 2025-12-03 16:23:49Z [hs-f689c7c4fae14d9e9a8e4ac1bc340a49.ctlplane-subnet]: DELETE_COMPLETE  state changed
2025-12-03 11:23:53.008623 | hotstack | 2025-12-03 16:23:49Z [hs-f689c7c4fae14d9e9a8e4ac1bc340a49.ctlplane-net]: DELETE_IN_PROGRESS  state changed
2025-12-03 11:23:53.008631 | hotstack | 2025-12-03 16:23:50Z [hs-f689c7c4fae14d9e9a8e4ac1bc340a49.storage-net]: DELETE_COMPLETE  state changed
2025-12-03 11:23:53.008639 | hotstack | 2025-12-03 16:23:50Z [hs-f689c7c4fae14d9e9a8e4ac1bc340a49.controller-machine-port]: DELETE_COMPLETE  state changed
2025-12-03 11:23:53.008650 | hotstack | 2025-12-03 16:23:50Z [hs-f689c7c4fae14d9e9a8e4ac1bc340a49.machine-subnet]: DELETE_IN_PROGRESS  state changed
2025-12-03 11:23:53.103850 | hotstack | 2025-12-03 16:23:50Z [hs-f689c7c4fae14d9e9a8e4ac1bc340a49.internal-api-net]: DELETE_COMPLETE  state changed
2025-12-03 11:23:53.103969 | hotstack | 2025-12-03 16:23:50Z [hs-f689c7c4fae14d9e9a8e4ac1bc340a49.ctlplane-net]: DELETE_COMPLETE  state changed
2025-12-03 11:23:53.103982 | hotstack | 2025-12-03 16:23:51Z [hs-f689c7c4fae14d9e9a8e4ac1bc340a49.machine-subnet]: DELETE_COMPLETE  state changed
2025-12-03 11:23:53.103990 | hotstack | 2025-12-03 16:23:51Z [hs-f689c7c4fae14d9e9a8e4ac1bc340a49.machine-net]: DELETE_IN_PROGRESS  state changed
2025-12-03 11:23:53.103998 | hotstack | 2025-12-03 16:23:51Z [hs-f689c7c4fae14d9e9a8e4ac1bc340a49.tenant-net]: DELETE_COMPLETE  state changed
2025-12-03 11:23:53.104006 | hotstack | 2025-12-03 16:23:52Z [hs-f689c7c4fae14d9e9a8e4ac1bc340a49.machine-net]: DELETE_COMPLETE  state changed
2025-12-03 11:23:53.104014 | hotstack | 2025-12-03 16:23:52Z [hs-f689c7c4fae14d9e9a8e4ac1bc340a49]: DELETE_COMPLETE  Stack DELETE completed successfully
2025-12-03 16:23:53.347886 | hotstack | changed
2025-12-03 16:23:53.367753 | 
2025-12-03 16:23:53.367946 | TASK [Print message that stack failed to delete]
2025-12-03 16:23:53.394900 | hotstack | skipping: Conditional result was False
2025-12-03 16:23:53.414257 | 
2025-12-03 16:23:53.414474 | TASK [Print message that stack deleted successfully]
2025-12-03 16:23:53.452443 | hotstack | ok: Stack deleted successfully.
2025-12-03 16:23:53.464385 | 
2025-12-03 16:23:53.464512 | TASK [Delete clouds.yaml]
2025-12-03 16:23:53.715551 | hotstack | Output suppressed because no_log was given
2025-12-03 16:23:53.754614 | 
2025-12-03 16:23:53.754723 | PLAY RECAP
2025-12-03 16:23:53.754777 | hotstack | ok: 21 changed: 8 unreachable: 0 failed: 0 skipped: 3 rescued: 2 ignored: 1
2025-12-03 16:23:53.754805 | 
2025-12-03 16:23:53.863952 | POST-RUN END RESULT_NORMAL: [trusted : review.rdoproject.org/config/playbooks/hotstack/post.yaml@master]
2025-12-03 16:23:53.866632 | POST-RUN START: [trusted : review.rdoproject.org/config/playbooks/base-minimal/post-ssh.yaml@master]
2025-12-03 16:23:54.470981 | 
2025-12-03 16:23:54.471101 | PLAY [all]
2025-12-03 16:23:54.490548 | 
2025-12-03 16:23:54.490633 | TASK [include_role : fetch-output]
2025-12-03 16:23:54.530306 | hotstack | ok
2025-12-03 16:23:54.549023 | 
2025-12-03 16:23:54.549103 | TASK [fetch-output : Set log path for multiple nodes]
2025-12-03 16:23:54.614485 | hotstack | skipping: Conditional result was False
2025-12-03 16:23:54.628034 | 
2025-12-03 16:23:54.628185 | TASK [fetch-output : Set log path for single node]
2025-12-03 16:23:54.676265 | hotstack | ok
2025-12-03 16:23:54.687771 | 
2025-12-03 16:23:54.687916 | LOOP [fetch-output : Ensure local output dirs]
2025-12-03 16:23:55.133158 | hotstack -> localhost | ok: "/var/lib/zuul/builds/f689c7c4fae14d9e9a8e4ac1bc340a49/work/logs"
2025-12-03 16:23:55.385313 | hotstack -> localhost | changed: "/var/lib/zuul/builds/f689c7c4fae14d9e9a8e4ac1bc340a49/work/artifacts"
2025-12-03 16:23:55.625724 | hotstack -> localhost | changed: "/var/lib/zuul/builds/f689c7c4fae14d9e9a8e4ac1bc340a49/work/docs"
2025-12-03 16:23:55.651874 | 
2025-12-03 16:23:55.652070 | LOOP [fetch-output : Collect logs, artifacts and docs]
2025-12-03 16:23:56.361779 | hotstack | changed:
2025-12-03 16:23:56.362177 | hotstack | .d..t...... ./
2025-12-03 16:23:56.362251 | hotstack | cd+++++++++ hotlogs/
2025-12-03 16:23:56.362296 | hotstack | cd+++++++++ hotlogs/data/
2025-12-03 16:23:56.362338 | hotstack | >f+++++++++ hotlogs/data/ironic_nodes.yaml
2025-12-03 16:23:56.362380 | hotstack | cd+++++++++ hotlogs/manifests/
2025-12-03 16:23:56.362420 | hotstack | cd+++++++++ hotlogs/ocp_cluster/
2025-12-03 16:23:56.362459 | hotstack | >f+++++++++ hotlogs/ocp_cluster/openshift_install.log
2025-12-03 16:23:56.362498 | hotstack | cd+++++++++ hotlogs/ocp_cluster/butane/
2025-12-03 16:23:56.362539 | hotstack | >f+++++++++ hotlogs/ocp_cluster/butane/90-master-disable-netifnames.bu
2025-12-03 16:23:56.362587 | hotstack | >f+++++++++ hotlogs/ocp_cluster/butane/90-master-enable-iscsi.bu
2025-12-03 16:23:56.362628 | hotstack | >f+++++++++ hotlogs/ocp_cluster/butane/91-master-enable-multipath.bu
2025-12-03 16:23:56.362697 | hotstack | >f+++++++++ hotlogs/ocp_cluster/butane/92-master-lv-cinder-volumes.bu
2025-12-03 16:23:56.362751 | hotstack | cd+++++++++ hotlogs/ocp_cluster/config-assets/
2025-12-03 16:23:56.362789 | hotstack | >f+++++++++ hotlogs/ocp_cluster/config-assets/95-etcd_config.yaml
2025-12-03 16:23:56.362835 | hotstack | >f+++++++++ hotlogs/ocp_cluster/config-assets/ovn_k8s_config.yaml
2025-12-03 16:23:56.362877 | hotstack | cd+++++++++ hotlogs/ocp_cluster/machine-configs/
2025-12-03 16:23:56.362915 | hotstack | >f+++++++++ hotlogs/ocp_cluster/machine-configs/90-master-disable-netifnames.yaml
2025-12-03 16:23:56.362952 | hotstack | >f+++++++++ hotlogs/ocp_cluster/machine-configs/90-master-enable-iscsi.yaml
2025-12-03 16:23:56.362988 | hotstack | >f+++++++++ hotlogs/ocp_cluster/machine-configs/91-master-enable-multipath.yaml
2025-12-03 16:23:56.363023 | hotstack | >f+++++++++ hotlogs/ocp_cluster/machine-configs/92-master-lv-cinder-volumes.yaml
2025-12-03 16:23:56.363090 | hotstack | changed: All items complete
2025-12-03 16:23:56.363134 | 
2025-12-03 16:23:56.911643 | hotstack | changed: .d..t...... ./
2025-12-03 16:23:57.455325 | hotstack | changed: .d..t...... ./
2025-12-03 16:23:57.490662 | 
2025-12-03 16:23:57.490885 | TASK [include_role : fetch-output-openshift]
2025-12-03 16:23:57.508976 | hotstack | skipping: Conditional result was False
2025-12-03 16:23:57.517008 | 
2025-12-03 16:23:57.517116 | LOOP [merge-output-to-logs : Move artifacts and docs to logs dir]
2025-12-03 16:23:57.958932 | hotstack -> localhost | ok: Item: artifacts Runtime: 0:00:00.014057
2025-12-03 16:23:58.252358 | hotstack -> localhost | ok: Item: docs Runtime: 0:00:00.013060
2025-12-03 16:23:58.303714 | 
2025-12-03 16:23:58.303874 | PLAY [all]
2025-12-03 16:23:58.323176 | 
2025-12-03 16:23:58.323272 | TASK [remove-build-sshkey : Remove the build SSH key from all nodes]
2025-12-03 16:23:58.942624 | hotstack | changed
2025-12-03 16:23:58.988458 | 
2025-12-03 16:23:58.988570 | PLAY RECAP
2025-12-03 16:23:58.988629 | hotstack | ok: 5 changed: 4 unreachable: 0 failed: 0 skipped: 2 rescued: 0 ignored: 0
2025-12-03 16:23:58.988660 | 
2025-12-03 16:23:59.089789 | POST-RUN END RESULT_NORMAL: [trusted : review.rdoproject.org/config/playbooks/base-minimal/post-ssh.yaml@master]
2025-12-03 16:23:59.095588 | POST-RUN START: [trusted : review.rdoproject.org/config/playbooks/base-minimal/post-logs.yaml@master]
2025-12-03 16:23:59.737774 | 
2025-12-03 16:23:59.737889 | PLAY [localhost]
2025-12-03 16:23:59.755555 | 
2025-12-03 16:23:59.755631 | TASK [generate-zuul-manifest : Generate Zuul manifest]
2025-12-03 16:24:00.179808 | localhost | changed
2025-12-03 16:24:00.184361 | 
2025-12-03 16:24:00.184434 | TASK [generate-zuul-manifest : Return Zuul manifest URL to Zuul]
2025-12-03 16:24:00.222137 | localhost | ok
2025-12-03 16:24:00.230081 | 
2025-12-03 16:24:00.230146 | TASK [add-fileserver : Create SSH private key tempfile]
2025-12-03 16:24:00.602338 | localhost | changed
2025-12-03 16:24:00.607930 | 
2025-12-03 16:24:00.607998 | TASK [add-fileserver : Create SSH private key from secret]
2025-12-03 16:24:01.314615 | localhost | changed
2025-12-03 16:24:01.319761 | 
2025-12-03 16:24:01.319830 | TASK [add-fileserver : Add fileserver ssh key]
2025-12-03 16:24:01.722743 | localhost | Identity added: /var/lib/zuul/builds/f689c7c4fae14d9e9a8e4ac1bc340a49/work/tmp/ansible.cyq4yns0 (/var/lib/zuul/builds/f689c7c4fae14d9e9a8e4ac1bc340a49/work/tmp/ansible.cyq4yns0)
2025-12-03 16:24:01.722959 | localhost | ok: Runtime: 0:00:00.012998
2025-12-03 16:24:01.727316 | 
2025-12-03 16:24:01.727384 | TASK [add-fileserver : Remove SSH private key from disk]
2025-12-03 16:24:02.000041 | localhost | ok: Runtime: 0:00:00.008352
2025-12-03 16:24:02.004864 | 
2025-12-03 16:24:02.004933 | TASK [add-fileserver : Add fileserver to inventory]
2025-12-03 16:24:02.084163 | localhost | changed
2025-12-03 16:24:02.088403 | 
2025-12-03 16:24:02.088467 | TASK [add-fileserver : Add fileserver server to known hosts]
2025-12-03 16:24:02.470642 | localhost | changed
2025-12-03 16:24:02.490563 | 
2025-12-03 16:24:02.490628 | PLAY [localhost]
2025-12-03 16:24:02.502240 | 
2025-12-03 16:24:02.502301 | TASK [Generate bulk log download script]
2025-12-03 16:24:02.545147 | localhost | ok
2025-12-03 16:24:02.557135 | 
2025-12-03 16:24:02.557204 | TASK [local-log-download : Check API endpoint is defined]
2025-12-03 16:24:02.596251 | localhost | ok: All assertions passed
2025-12-03 16:24:02.600811 | 
2025-12-03 16:24:02.600875 | TASK [local-log-download : Create download script]
2025-12-03 16:24:03.072302 | localhost -> localhost | changed
2025-12-03 16:24:03.083983 | 
2025-12-03 16:24:03.084058 | TASK [Register quick-download link]
2025-12-03 16:24:03.104789 | localhost | ok
2025-12-03 16:24:03.124832 | 
2025-12-03 16:24:03.124905 | PLAY [logserver.rdoproject.org]
2025-12-03 16:24:03.136574 | 
2025-12-03 16:24:03.136639 | TASK [Set zuul-log-path fact]
2025-12-03 16:24:03.154318 | logserver.rdoproject.org | ok
2025-12-03 16:24:03.166087 | 
2025-12-03 16:24:03.166161 | TASK [set-zuul-log-path-fact : Set log path for a build]
2025-12-03 16:24:03.204984 | logserver.rdoproject.org | ok
2025-12-03 16:24:03.214445 | 
2025-12-03 16:24:03.214514 | TASK [upload-logs : Create log directories]
2025-12-03 16:24:04.061947 | logserver.rdoproject.org | changed
2025-12-03 16:24:04.068569 | 
2025-12-03 16:24:04.068737 | TASK [upload-logs : Ensure logs are readable before uploading]
2025-12-03 16:24:04.370644 | logserver.rdoproject.org -> localhost | ok: Runtime: 0:00:00.008905
2025-12-03 16:24:04.381122 | 
2025-12-03 16:24:04.381291 | TASK [upload-logs : Upload logs to log server]
2025-12-03 16:24:05.137556 | logserver.rdoproject.org | Output suppressed because no_log was given
2025-12-03 16:24:05.144298 | 
2025-12-03 16:24:05.144450 | LOOP [upload-logs : Compress console log and json output]
2025-12-03 16:24:05.213176 | logserver.rdoproject.org | skipping: Conditional result was False
2025-12-03 16:24:05.224745 | logserver.rdoproject.org | skipping: Conditional result was False
2025-12-03 16:24:05.239383 | 
2025-12-03 16:24:05.239530 | LOOP [upload-logs : Upload compressed console log and json output]
2025-12-03 16:24:05.288995 | logserver.rdoproject.org | skipping: Conditional result was False
2025-12-03 16:24:05.289382 | 
2025-12-03 16:24:05.292340 | logserver.rdoproject.org | skipping: Conditional result was False
2025-12-03 16:24:05.305886 | 
2025-12-03 16:24:05.306032 | LOOP [upload-logs : Upload console log and json output]