2026-03-09 14:00:21.868572 | Job console starting...
2026-03-09 14:00:21.881031 | Updating repositories
2026-03-09 14:00:21.915382 | Preparing job workspace
2026-03-09 14:00:28.045903 | Running Ansible setup...
2026-03-09 14:00:31.893103 | PRE-RUN START: [trusted : review.rdoproject.org/config/playbooks/base-minimal/pre.yaml@master]
2026-03-09 14:00:32.565445 | 
2026-03-09 14:00:32.565588 | PLAY [localhost]
2026-03-09 14:00:32.575203 | 
2026-03-09 14:00:32.575280 | TASK [Gathering Facts]
2026-03-09 14:00:33.545120 | localhost | ok
2026-03-09 14:00:33.558553 | 
2026-03-09 14:00:33.558637 | TASK [log-inventory : Ensure Zuul Ansible directory exists]
2026-03-09 14:00:34.032399 | localhost -> localhost | changed
2026-03-09 14:00:34.054117 | 
2026-03-09 14:00:34.054325 | TASK [log-inventory : Copy ansible inventory to logs dir]
2026-03-09 14:00:34.928527 | localhost -> localhost | changed
2026-03-09 14:00:34.936872 | 
2026-03-09 14:00:34.936951 | TASK [Setup log path fact]
2026-03-09 14:00:34.983259 | localhost | ok
2026-03-09 14:00:35.007329 | 
2026-03-09 14:00:35.007495 | TASK [set-zuul-log-path-fact : Set log path for a build]
2026-03-09 14:00:35.038498 | localhost | ok
2026-03-09 14:00:35.048814 | 
2026-03-09 14:00:35.048912 | TASK [emit-job-header : Print job information]
2026-03-09 14:00:35.077206 | # Job Information
2026-03-09 14:00:35.077641 | Ansible Version: 2.15.12
2026-03-09 14:00:35.077682 | Job: hotstack-sno-1-bm
2026-03-09 14:00:35.077704 | Pipeline: github-check
2026-03-09 14:00:35.077736 | Executor: ze03.softwarefactory-project.io
2026-03-09 14:00:35.077759 | Triggered by: https://github.com/openstack-k8s-operators/ironic-operator/pull/686
2026-03-09 14:00:35.077778 | Log URL (when completed): https://logserver.rdoproject.org/5ad/rdoproject.org/5ad27496488549c98262fbfcd2662052/
2026-03-09 14:00:35.077796 | Event ID: bb8f5de0-1bbd-11f1-8322-989e03fea142
2026-03-09 14:00:35.081364 | 
2026-03-09 14:00:35.081426 | LOOP [emit-job-header : Print node information]
2026-03-09 14:00:35.179799 | localhost | ok:
2026-03-09 14:00:35.179965 | localhost | # Node Information
2026-03-09 14:00:35.179991 | localhost | Inventory Hostname: hotstack
2026-03-09 14:00:35.180014 | localhost | Hostname: np0005642478
2026-03-09 14:00:35.180033 | localhost | Username: zuul
2026-03-09 14:00:35.180053 | localhost | Distro: CentOS 9
2026-03-09 14:00:35.180071 | localhost | Provider: vexxhost-nodepool-tripleo
2026-03-09 14:00:35.180088 | localhost | Region: RegionOne
2026-03-09 14:00:35.180104 | localhost | Label: cloud-centos-9-stream-tripleo-vexxhost
2026-03-09 14:00:35.180121 | localhost | Product Name: OpenStack Nova
2026-03-09 14:00:35.180137 | localhost | Interface IP: 38.102.83.210
2026-03-09 14:00:35.205768 | 
2026-03-09 14:00:35.205878 | PLAY [all]
2026-03-09 14:00:35.212208 | 
2026-03-09 14:00:35.212270 | TASK [Gather network facts]
2026-03-09 14:00:35.640980 | hotstack | ok
2026-03-09 14:00:35.657960 | 
2026-03-09 14:00:35.658046 | TASK [include_role : start-zuul-console]
2026-03-09 14:00:35.680585 | hotstack | ok
2026-03-09 14:00:35.696713 | 
2026-03-09 14:00:35.696814 | TASK [start-zuul-console : Start zuul_console daemon.]
2026-03-09 14:00:36.142474 | hotstack | ok
2026-03-09 14:00:36.151259 | 
2026-03-09 14:00:36.151333 | TASK [include_role : add-build-sshkey]
2026-03-09 14:00:36.170358 | hotstack | ok
2026-03-09 14:00:36.186241 | 
2026-03-09 14:00:36.186333 | TASK [add-build-sshkey : Check to see if ssh key was already created for this build]
2026-03-09 14:00:36.417342 | hotstack -> localhost | ok
2026-03-09 14:00:36.422862 | 
2026-03-09 14:00:36.422928 | TASK [add-build-sshkey : Create a new key in workspace based on build UUID]
2026-03-09 14:00:36.451797 | hotstack | ok
2026-03-09 14:00:36.466493 | hotstack | included: /var/lib/zuul/builds/5ad27496488549c98262fbfcd2662052/trusted/project_1/opendev.org/zuul/zuul-jobs/roles/add-build-sshkey/tasks/create-key-and-replace.yaml
2026-03-09 14:00:36.472459 | 
2026-03-09 14:00:36.472519 | TASK [add-build-sshkey : Create Temp SSH key]
2026-03-09 14:00:37.261974 | hotstack -> localhost | Generating public/private rsa key pair.
2026-03-09 14:00:37.262210 | hotstack -> localhost | Your identification has been saved in /var/lib/zuul/builds/5ad27496488549c98262fbfcd2662052/work/5ad27496488549c98262fbfcd2662052_id_rsa.
2026-03-09 14:00:37.262243 | hotstack -> localhost | Your public key has been saved in /var/lib/zuul/builds/5ad27496488549c98262fbfcd2662052/work/5ad27496488549c98262fbfcd2662052_id_rsa.pub.
2026-03-09 14:00:37.262265 | hotstack -> localhost | The key fingerprint is:
2026-03-09 14:00:37.262285 | hotstack -> localhost | SHA256:/ekb1LYkqgEoKGRuFb33a76oytDp2KGyOjJ7rwx/AWo zuul-build-sshkey
2026-03-09 14:00:37.262304 | hotstack -> localhost | The key's randomart image is:
2026-03-09 14:00:37.262321 | hotstack -> localhost | +---[RSA 3072]----+
2026-03-09 14:00:37.262339 | hotstack -> localhost | |   ..            |
2026-03-09 14:00:37.262357 | hotstack -> localhost | |    ..           |
2026-03-09 14:00:37.262374 | hotstack -> localhost | | o .  .          |
2026-03-09 14:00:37.262391 | hotstack -> localhost | |+.o  o . .   .   |
2026-03-09 14:00:37.262407 | hotstack -> localhost | |o+... o S . o +  |
2026-03-09 14:00:37.262424 | hotstack -> localhost | |+E..o  . . + = . |
2026-03-09 14:00:37.262440 | hotstack -> localhost | |o. + .  . o + .  |
2026-03-09 14:00:37.262457 | hotstack -> localhost | |=+O o   .= . .   |
2026-03-09 14:00:37.262473 | hotstack -> localhost | |BO+Oo...+o. o.   |
2026-03-09 14:00:37.262490 | hotstack -> localhost | +----[SHA256]-----+
2026-03-09 14:00:37.262540 | hotstack -> localhost | ok: Runtime: 0:00:00.259770
2026-03-09 14:00:37.268409 | 
2026-03-09 14:00:37.268470 | TASK [add-build-sshkey : Remote setup ssh keys (linux)]
2026-03-09 14:00:37.298451 | hotstack | ok
2026-03-09 14:00:37.308429 | hotstack | included: /var/lib/zuul/builds/5ad27496488549c98262fbfcd2662052/trusted/project_1/opendev.org/zuul/zuul-jobs/roles/add-build-sshkey/tasks/remote-linux.yaml
2026-03-09 14:00:37.316264 | 
2026-03-09 14:00:37.316322 | TASK [add-build-sshkey : Remove previously added zuul-build-sshkey]
2026-03-09 14:00:37.341330 | hotstack | skipping: Conditional result was False
2026-03-09 14:00:37.348418 | 
2026-03-09 14:00:37.348483 | TASK [add-build-sshkey : Enable access via build key on all nodes]
2026-03-09 14:00:37.783079 | hotstack | changed
2026-03-09 14:00:37.788246 | 
2026-03-09 14:00:37.788312 | TASK [add-build-sshkey : Make sure user has a .ssh]
2026-03-09 14:00:38.077138 | hotstack | ok
2026-03-09 14:00:38.091862 | 
2026-03-09 14:00:38.092022 | TASK [add-build-sshkey : Install build private key as SSH key on all nodes]
2026-03-09 14:00:38.791453 | hotstack | changed
2026-03-09 14:00:38.797383 | 
2026-03-09 14:00:38.797483 | TASK [add-build-sshkey : Install build public key as SSH key on all nodes]
2026-03-09 14:00:39.459691 | hotstack | changed
2026-03-09 14:00:39.469591 | 
2026-03-09 14:00:39.469708 | TASK [add-build-sshkey : Remote setup ssh keys (windows)]
2026-03-09 14:00:39.488846 | hotstack | skipping: Conditional result was False
2026-03-09 14:00:39.500167 | 
2026-03-09 14:00:39.500302 | TASK [remove-zuul-sshkey : Remove master key from local agent]
2026-03-09 14:00:39.893517 | hotstack -> localhost | changed
2026-03-09 14:00:39.903057 | 
2026-03-09 14:00:39.903123 | TASK [add-build-sshkey : Add back temp key]
2026-03-09 14:00:40.221848 | hotstack -> localhost | Identity added: /var/lib/zuul/builds/5ad27496488549c98262fbfcd2662052/work/5ad27496488549c98262fbfcd2662052_id_rsa (zuul-build-sshkey)
2026-03-09 14:00:40.222066 | hotstack -> localhost | ok: Runtime: 0:00:00.009064
2026-03-09 14:00:40.227752 | 
2026-03-09 14:00:40.227823 | TASK [add-build-sshkey : Verify we can still SSH to all nodes]
2026-03-09 14:00:40.606205 | hotstack | ok
2026-03-09 14:00:40.612607 | 
2026-03-09 14:00:40.612695 | TASK [add-build-sshkey : Verify we can still SSH to all nodes (windows)]
2026-03-09 14:00:40.637602 | hotstack | skipping: Conditional result was False
2026-03-09 14:00:40.684908 | 
2026-03-09 14:00:40.685056 | TASK [include_role : validate-host]
2026-03-09 14:00:40.705689 | hotstack | ok
2026-03-09 14:00:40.730691 | 
2026-03-09 14:00:40.730794 | TASK [validate-host : Define zuul_info_dir fact]
2026-03-09 14:00:40.781489 | hotstack | ok
2026-03-09 14:00:40.787675 | 
2026-03-09 14:00:40.787776 | TASK [validate-host : Ensure Zuul Ansible directory exists]
2026-03-09 14:00:41.037391 | hotstack -> localhost | ok
2026-03-09 14:00:41.049835 | 
2026-03-09 14:00:41.049912 | TASK [validate-host : Collect information about the host]
2026-03-09 14:00:41.766163 | hotstack | ok
2026-03-09 14:00:41.778282 | 
2026-03-09 14:00:41.778413 | TASK [validate-host : Sanitize hostname]
2026-03-09 14:00:41.828035 | hotstack | ok
2026-03-09 14:00:41.833520 | 
2026-03-09 14:00:41.833584 | TASK [validate-host : Write out all ansible variables/facts known for each host]
2026-03-09 14:00:42.372419 | hotstack -> localhost | changed
2026-03-09 14:00:42.378296 | 
2026-03-09 14:00:42.378370 | TASK [validate-host : Collect information about zuul worker]
2026-03-09 14:00:42.783837 | hotstack | ok
2026-03-09 14:00:42.789358 | 
2026-03-09 14:00:42.789421 | TASK [validate-host : Write out all zuul information for each host]
2026-03-09 14:00:43.291079 | hotstack -> localhost | changed
2026-03-09 14:00:43.311214 | 
2026-03-09 14:00:43.311345 | TASK [include_role : prepare-workspace-openshift]
2026-03-09 14:00:43.337575 | hotstack | skipping: Conditional result was False
2026-03-09 14:00:43.344347 | 
2026-03-09 14:00:43.344433 | TASK [include_role : remove-zuul-sshkey]
2026-03-09 14:00:43.368822 | hotstack | skipping: Conditional result was False
2026-03-09 14:00:43.376394 | 
2026-03-09 14:00:43.376484 | LOOP [ensure-output-dirs : Empty Zuul Output directories by removing them]
2026-03-09 14:00:43.632937 | hotstack | ok: "logs"
2026-03-09 14:00:43.633373 | hotstack | ok: All items complete
2026-03-09 14:00:43.633433 | 
2026-03-09 14:00:43.869630 | hotstack | ok: "artifacts"
2026-03-09 14:00:44.047959 | hotstack | ok: "docs"
2026-03-09 14:00:44.070461 | 
2026-03-09 14:00:44.070702 | LOOP [ensure-output-dirs : Ensure Zuul Output directories exist]
2026-03-09 14:00:44.326544 | hotstack | changed: "logs"
2026-03-09 14:00:44.531763 | hotstack | changed: "artifacts"
2026-03-09 14:00:44.735388 | hotstack | changed: "docs"
2026-03-09 14:00:44.782410 | 
2026-03-09 14:00:44.782518 | PLAY RECAP
2026-03-09 14:00:44.782574 | hotstack | ok: 22 changed: 9 unreachable: 0 failed: 0 skipped: 5 rescued: 0 ignored: 0
2026-03-09 14:00:44.782607 | localhost | ok: 6 changed: 2 unreachable: 0 failed: 0 skipped: 0 rescued: 0 ignored: 0
2026-03-09 14:00:44.782631 | 
2026-03-09 14:00:44.895021 | PRE-RUN END RESULT_NORMAL: [trusted : review.rdoproject.org/config/playbooks/base-minimal/pre.yaml@master]
2026-03-09 14:00:44.899257 | PRE-RUN START: [trusted : review.rdoproject.org/config/playbooks/hotstack/pre_stage1.yaml@master]
2026-03-09 14:00:45.523711 | 
2026-03-09 14:00:45.523850 | PLAY [Create necessary files to deploy hotstack scenario]
2026-03-09 14:00:45.544204 | 
2026-03-09 14:00:45.544298 | TASK [prepare-workspace : Start zuul_console daemon.]
2026-03-09 14:00:45.939627 | hotstack | ok
2026-03-09 14:00:45.947997 | 
2026-03-09 14:00:45.948097 | TASK [prepare-workspace : Synchronize src repos to workspace directory.]
2026-03-09 14:00:49.217460 | hotstack | Output suppressed because no_log was given
2026-03-09 14:00:49.227683 | 
2026-03-09 14:00:49.227772 | LOOP [add-authorized-keys : Enable access via build key on all nodes]
2026-03-09 14:00:49.656464 | hotstack | changed:
2026-03-09 14:00:49.656782 | hotstack | {
2026-03-09 14:00:49.656852 | hotstack |   "public_key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCINPOs6XsfMVQMqjmiduT3Bxhogy7MbAFOWlR+9LlKkTW1kV6glsCXY5svtggd2zRtc40UWalHaj9jC+yEkZdI35h2PkK+Sww52u4nczqgNpIFNt7Dm2xtYKbTolAV9jSeNkRlGMCq9EBzQlwiV0HkHx1CdNR/zzXbyNZLmOybFkXqR150H01kUhdv0v6AMWTDZjMEWO/PaWCOgzZhH9ztXjy+rwa7pdaI4+0Pb9Kib+d8/IrgNsLB/sCazqD+9/COI6AHqzTDS1aU9/gMQrUN1ronzMEQhzzOuoI8WeLqU++j+0dT2UWVPlQqoln+RYu7dUn/DSZDrsYenudT5+P"
2026-03-09 14:00:49.656899 | hotstack | }
2026-03-09 14:00:49.684041 | 
2026-03-09 14:00:49.684226 | TASK [Assert cloud name and cloud secret]
2026-03-09 14:00:49.738921 | hotstack | Output suppressed because no_log was given
2026-03-09 14:00:49.746525 | 
2026-03-09 14:00:49.746633 | TASK [Ensure openstack release repo is installed]
2026-03-09 14:00:54.075955 | hotstack | changed
2026-03-09 14:00:54.087411 | 
2026-03-09 14:00:54.087581 | TASK [Install openstack client packages]
2026-03-09 14:01:08.302462 | hotstack | changed
2026-03-09 14:01:08.307774 | 
2026-03-09 14:01:08.307833 | TASK [Ensure HotStack work dir]
2026-03-09 14:01:08.739369 | hotstack | changed
2026-03-09 14:01:08.756372 | 
2026-03-09 14:01:08.756575 | LOOP [Ensure etc directories]
2026-03-09 14:01:09.065710 | hotstack | changed: "/etc/openstack"
2026-03-09 14:01:09.066188 | hotstack | changed: All items complete
2026-03-09 14:01:09.066254 | 
2026-03-09 14:01:09.268874 | hotstack | changed: "/etc/hotstack"
2026-03-09 14:01:09.289540 | 
2026-03-09 14:01:09.289682 | TASK [Ensure .ssh dir exists]
2026-03-09 14:01:09.514380 | hotstack | Output suppressed because no_log was given
2026-03-09 14:01:09.527941 | 
2026-03-09 14:01:09.528089 | TASK [Ensure ssh keypair exists]
2026-03-09 14:01:20.093746 | hotstack | Output suppressed because no_log was given
2026-03-09 14:01:20.099316 | 
2026-03-09 14:01:20.099380 | TASK [Generate CA cert file]
2026-03-09 14:01:20.143653 | hotstack | skipping: Conditional result was False
2026-03-09 14:01:20.151139 | 
2026-03-09 14:01:20.151224 | TASK [Update CA certs]
2026-03-09 14:01:20.688111 | hotstack | skipping: Conditional result was False
2026-03-09 14:01:20.700978 | 
2026-03-09 14:01:20.701116 | TASK [Set _auth_type fact]
2026-03-09 14:01:20.763421 | hotstack | Output suppressed because no_log was given
2026-03-09 14:01:20.775113 | 
2026-03-09 14:01:20.775245 | TASK [Write clouds.yaml - user/password]
2026-03-09 14:01:21.802583 | hotstack | Output suppressed because no_log was given
2026-03-09 14:01:21.814451 | 
2026-03-09 14:01:21.814587 | TASK [Create application credential for sushy-emulator]
2026-03-09 14:01:24.371333 | hotstack | Output suppressed because no_log was given
2026-03-09 14:01:24.384116 | 
2026-03-09 14:01:24.384285 | TASK [Read application credential from file]
2026-03-09 14:01:24.723307 | hotstack | Output suppressed because no_log was given
2026-03-09 14:01:24.730718 | 
2026-03-09 14:01:24.730827 | TASK [Delete temporary application credential file]
2026-03-09 14:01:25.012419 | hotstack | Output suppressed because no_log was given
2026-03-09 14:01:25.028578 | 
2026-03-09 14:01:25.028802 | TASK [Assert application credential id and secret]
2026-03-09 14:01:25.130590 | hotstack | Output suppressed because no_log was given
2026-03-09 14:01:25.137437 | 
2026-03-09 14:01:25.137519 | TASK [Write new clouds.yaml with application credentials]
2026-03-09 14:01:25.999250 | hotstack | Output suppressed because no_log was given
2026-03-09 14:01:26.020037 | 
2026-03-09 14:01:26.020239 | TASK [Write hotstack cloud-secret.yaml]
2026-03-09 14:01:26.743660 | hotstack | Output suppressed because no_log was given
2026-03-09 14:01:26.750879 | 
2026-03-09 14:01:26.750976 | TASK [Write clouds.yaml with application credentials]
2026-03-09 14:01:26.787548 | 
2026-03-09 14:01:26.787748 | TASK [Write hotstack cloud-secret.yaml]
2026-03-09 14:01:26.828905 | 
2026-03-09 14:01:26.829044 | PLAY RECAP
2026-03-09 14:01:26.829101 | hotstack | ok: 18 changed: 11 unreachable: 0 failed: 0 skipped: 4 rescued: 0 ignored: 0
2026-03-09 14:01:26.829150 | 
2026-03-09 14:01:26.926722 | PRE-RUN END RESULT_NORMAL: [trusted : review.rdoproject.org/config/playbooks/hotstack/pre_stage1.yaml@master]
2026-03-09 14:01:26.932891 | PRE-RUN START: [trusted : review.rdoproject.org/config/playbooks/hotstack/pre_stage2.yaml@master]
2026-03-09 14:01:27.503183 | 
2026-03-09 14:01:27.503309 | PLAY [Create hotstack scenario]
2026-03-09 14:01:27.526074 | 
2026-03-09 14:01:27.526196 | TASK [Assert scenario is defined]
2026-03-09 14:01:27.613344 | hotstack | ok: All assertions passed
2026-03-09 14:01:27.626002 | 
2026-03-09 14:01:27.626105 | TASK [Fetch bootstrap vars from remote to zuul executor]
2026-03-09 14:01:28.054837 | hotstack | Output suppressed because no_log was given
2026-03-09 14:01:28.060135 | 
2026-03-09 14:01:28.060205 | TASK [Load bootstrap vars with template resolution]
2026-03-09 14:01:28.089619 | hotstack | Output suppressed because no_log was given
2026-03-09 14:01:28.094420 | 
2026-03-09 14:01:28.094486 | TASK [Fetch automation vars from remote to zuul executor]
2026-03-09 14:01:28.439903 | hotstack | Output suppressed because no_log was given
2026-03-09 14:01:28.452035 | 
2026-03-09 14:01:28.452177 | TASK [Load automation vars with template resolution]
2026-03-09 14:01:28.543218 | hotstack | Output suppressed because no_log was given
2026-03-09 14:01:28.554527 | 
2026-03-09 14:01:28.554658 | TASK [Slurp zuul-build-sshkey from file]
2026-03-09 14:01:28.920499 | hotstack | Output suppressed because no_log was given
2026-03-09 14:01:28.934036 | 
2026-03-09 14:01:28.934254 | TASK [Set up infrastructure (Heat stack)]
2026-03-09 14:01:28.980884 | hotstack | Output suppressed because no_log was given
2026-03-09 14:01:29.041091 | 
2026-03-09 14:01:29.041243 | TASK [dataplane_ssh_keys : Ensure Dataplane ssh keypair exists]
2026-03-09 14:01:29.691678 | hotstack | changed
2026-03-09 14:01:29.722603 | 
2026-03-09 14:01:29.722825 | TASK [dataplane_ssh_keys : Ensure Nova migration ssh keypair exists]
2026-03-09 14:01:30.084666 | hotstack | changed
2026-03-09 14:01:30.096852 | 
2026-03-09 14:01:30.097064 | TASK [dataplane_ssh_keys : Set SSH public key facts]
2026-03-09 14:01:30.135838 | hotstack | ok
2026-03-09 14:01:30.148183 | 
2026-03-09 14:01:30.148311 | TASK [dataplane_ssh_keys : Store SSH public key facts in file]
2026-03-09 14:01:30.953542 | hotstack | changed
2026-03-09 14:01:30.966821 | 
2026-03-09 14:01:30.966956 | TASK [heat_stack : Assert config is defined]
2026-03-09 14:01:31.082066 | hotstack | ok: All assertions passed
2026-03-09 14:01:31.092078 | 
2026-03-09 14:01:31.092224 | TASK [heat_stack : Get latest snapset and update stack parameters]
2026-03-09 14:01:31.149063 | hotstack | skipping: Conditional result was False
2026-03-09 14:01:31.157845 | 
2026-03-09 14:01:31.157956 | TASK [heat_stack : Debug latest snapset information]
2026-03-09 14:01:31.224206 | hotstack | skipping: Conditional result was False
2026-03-09 14:01:31.236212 | 
2026-03-09 14:01:31.236362 | LOOP [heat_stack : Create tar archives and compress]
2026-03-09 14:01:31.259608 | 
2026-03-09 14:01:31.259800 | TASK [heat_stack : Create stack]
2026-03-09 14:02:53.015998 | hotstack | changed
2026-03-09 14:02:53.043459 | 
2026-03-09 14:02:53.043665 | TASK [heat_stack : Wait for stack to complete]
2026-03-09 14:02:54.972128 | hotstack | ok
2026-03-09 14:02:54.986253 | 
2026-03-09 14:02:54.986389 | TASK [heat_stack : Stop if stack not complete]
2026-03-09 14:02:55.024704 | hotstack | skipping: Conditional result was False
2026-03-09 14:02:55.038893 | 
2026-03-09 14:02:55.039066 | TASK [heat_stack : Get stack outputs]
2026-03-09 14:02:57.186753 | hotstack | changed
2026-03-09 14:02:57.203643 | 
2026-03-09 14:02:57.203902 | TASK [heat_stack : Set stack outputs facts]
2026-03-09 14:02:57.248620 | hotstack | ok
2026-03-09 14:02:57.257608 | 
2026-03-09 14:02:57.257796 | TASK [heat_stack : Store stack outputs in file]
2026-03-09 14:02:57.936868 | hotstack | changed
2026-03-09 14:02:57.958993 | 
2026-03-09 14:02:57.959142 | TASK [Fetch stack outputs from remote to zuul executor]
2026-03-09 14:02:58.271937 | hotstack | Output suppressed because no_log was given
2026-03-09 14:02:58.282711 | 
2026-03-09 14:02:58.282892 | TASK [Load stack output vars with template resolution]
2026-03-09 14:02:58.320336 | hotstack | Output suppressed because no_log was given
2026-03-09 14:02:58.330773 | 
2026-03-09 14:02:58.330940 | TASK [Fetch cloud secrets from remote to zuul executor]
2026-03-09 14:02:58.615035 | hotstack | Output suppressed because no_log was given
2026-03-09 14:02:58.629011 | 
2026-03-09 14:02:58.629184 | TASK [Load hotstack cloud secret vars with template resolution]
2026-03-09 14:02:58.677387 | hotstack | Output suppressed because no_log was given
2026-03-09 14:02:58.688375 | 
2026-03-09 14:02:58.688506 | TASK [Add Hotstack controller-0 to ansible inventory]
2026-03-09 14:02:58.736160 | hotstack | Output suppressed because no_log was given
2026-03-09 14:02:58.747963 | 
2026-03-09 14:02:58.748097 | TASK [Wait for Hotstack controller-0 to be ready]
2026-03-09 14:03:05.457937 | hotstack -> 38.102.83.163 | Output suppressed because no_log was given
2026-03-09 14:03:05.474861 | 
2026-03-09 14:03:05.475049 | TASK [Start Zuul console on Hotstack controller-0]
2026-03-09 14:03:05.520495 | hotstack | Output suppressed because no_log was given
2026-03-09 14:03:05.630448 | 
2026-03-09 14:03:05.630674 | TASK [start-zuul-console : Start zuul_console daemon.]
2026-03-09 14:03:06.508190 | hotstack -> 38.102.83.163 | ok
2026-03-09 14:03:06.519316 | 
2026-03-09 14:03:06.519452 | TASK [Prepare the controller node]
2026-03-09 14:03:06.538878 | hotstack | Output suppressed because no_log was given
2026-03-09 14:03:06.585587 | 
2026-03-09 14:03:06.585702 | TASK [controller : Assert config is defined]
2026-03-09 14:03:06.661138 | hotstack | ok: All assertions passed
2026-03-09 14:03:06.668603 | 
2026-03-09 14:03:06.668754 | TASK [controller : Add controller-0 to the Ansible inventory]
2026-03-09 14:03:06.707679 | hotstack | ok
2026-03-09 14:03:06.713468 | 
2026-03-09 14:03:06.713531 | TASK [controller : Read Dataplane SSH private key content]
2026-03-09 14:03:06.905621 | hotstack | ok
2026-03-09 14:03:06.910675 | 
2026-03-09 14:03:06.910759 | TASK [controller : Read Nova migration SSH private key content]
2026-03-09 14:03:07.122148 | hotstack | ok
2026-03-09 14:03:07.127155 | 
2026-03-09 14:03:07.127221 | TASK [controller : Wait for controller-0 to be ready]
2026-03-09 14:03:07.982706 | hotstack -> 38.102.83.163 | ok
2026-03-09 14:03:07.997379 | 
2026-03-09 14:03:07.997483 | LOOP [controller : Ensure directories exists]
2026-03-09 14:03:08.829506 | hotstack -> 38.102.83.163 | changed: "/home/zuul/data"
2026-03-09 14:03:08.830120 | 
2026-03-09 14:03:09.535306 | hotstack -> 38.102.83.163 | changed: "/home/zuul/bin"
2026-03-09 14:03:09.569100 | 
2026-03-09 14:03:09.569306 | TASK [controller : Setup NFS server]
2026-03-09 14:03:09.665789 | hotstack | ok
2026-03-09 14:03:09.697066 | hotstack | included: /var/lib/zuul/builds/5ad27496488549c98262fbfcd2662052/trusted/project_3/github.com/openstack-k8s-operators/hotstack/roles/controller/tasks/setup_nfs.yml
2026-03-09 14:03:09.712178 | 
2026-03-09 14:03:09.712397 | TASK [controller : Install NFS server packages]
2026-03-09 14:03:14.029636 | hotstack -> 38.102.83.163 | ok: Nothing to do
2026-03-09 14:03:14.044365 | 
2026-03-09 14:03:14.044514 | LOOP [controller : Create NFS export directories]
2026-03-09 14:03:14.386217 | hotstack -> 38.102.83.163 | changed:
2026-03-09 14:03:14.386617 | hotstack -> 38.102.83.163 | {
2026-03-09 14:03:14.386710 | hotstack -> 38.102.83.163 |   "mode": "0777",
2026-03-09 14:03:14.386798 | hotstack -> 38.102.83.163 |   "path": "/export/nova-console-recordings"
2026-03-09 14:03:14.386842 | hotstack -> 38.102.83.163 | }
2026-03-09 14:03:14.405396 | 
2026-03-09 14:03:14.405628 | TASK [controller : Configure NFS exports]
2026-03-09 14:03:15.200422 | hotstack -> 38.102.83.163 | changed
2026-03-09 14:03:15.218716 | 
2026-03-09 14:03:15.218891 | TASK [controller : Enable and start NFS server]
2026-03-09 14:03:16.839845 | hotstack -> 38.102.83.163 | changed
2026-03-09 14:03:16.849005 | 
2026-03-09 14:03:16.849086 | TASK [controller : Reload NFS exports]
2026-03-09 14:03:17.384000 | hotstack -> 38.102.83.163 | ok
2026-03-09 14:03:17.393138 | 
2026-03-09 14:03:17.393238 | LOOP [controller : Add common functions to the bin_dir]
2026-03-09 14:03:18.231529 | hotstack -> 38.102.83.163 | changed: "bin/hotstack-common-funcs.sh"
2026-03-09 14:03:18.231841 | hotstack -> 38.102.83.163 | changed: All items complete
2026-03-09 14:03:18.231873 | 
2026-03-09 14:03:18.249067 | 
2026-03-09 14:03:18.249238 | LOOP [controller : Add bin utilities to the bin_dir]
2026-03-09 14:03:19.351189 | hotstack -> 38.102.83.163 | changed: "bin/hotstack-approve-install-plan"
2026-03-09 14:03:20.399860 | hotstack -> 38.102.83.163 | changed: "bin/hotstack-collect-tempest-logs"
2026-03-09 14:03:21.452637 | hotstack -> 38.102.83.163 | changed: "bin/hotstack-leader-election-tune"
2026-03-09 14:03:22.069409 | hotstack -> 38.102.83.163 | changed: "bin/hotstack-nova-discover-hosts"
2026-03-09 14:03:22.686719 | hotstack -> 38.102.83.163 | changed: "bin/hotstack-snapset"
2026-03-09 14:03:23.293567 | hotstack -> 38.102.83.163 | changed: "bin/hotstack-openstack-version-patch"
2026-03-09 14:03:23.886890 | hotstack -> 38.102.83.163 | changed: "bin/hotstack-wait-for-bmh"
2026-03-09 14:03:23.901523 | 
2026-03-09 14:03:23.901671 | TASK [controller : Write ansible inventory to file on controller-0]
2026-03-09 14:03:23.967585 | hotstack | skipping: Conditional result was False
2026-03-09 14:03:23.974420 | 
2026-03-09 14:03:23.974500 | TASK [controller : Copy Dataplane ssh private key to controller]
2026-03-09 14:03:24.592962 | hotstack -> 38.102.83.163 | changed
2026-03-09 14:03:24.610702 | 
2026-03-09 14:03:24.611004 | TASK [controller : Copy Dataplane ssh public key to controller]
2026-03-09 14:03:25.253103 | hotstack -> 38.102.83.163 | changed
2026-03-09 14:03:25.267848 | 
2026-03-09 14:03:25.268020 | TASK [controller : Copy Nova migrate ssh private key to controller]
2026-03-09 14:03:25.939969 | hotstack -> 38.102.83.163 | changed
2026-03-09 14:03:25.949845 | 
2026-03-09 14:03:25.949962 | TASK [controller : Copy Nova migrate ssh public key to controller]
2026-03-09 14:03:26.611059 | hotstack -> 38.102.83.163 | changed
2026-03-09 14:03:26.620933 | 
2026-03-09 14:03:26.621055 | TASK [controller : Copy dataplane authorized keys to controller]
2026-03-09 14:03:27.439001 | hotstack -> 38.102.83.163 | changed
2026-03-09 14:03:27.509794 | 
2026-03-09 14:03:27.509943 | TASK [controller : Ensure cloud config directory exists]
2026-03-09 14:03:27.842287 | hotstack -> 38.102.83.163 | changed
2026-03-09 14:03:27.858104 | 
2026-03-09 14:03:27.858300 | TASK [controller : Template the clouds.yaml]
2026-03-09 14:03:28.582605 | hotstack -> 38.102.83.163 | Output suppressed because no_log was given
2026-03-09 14:03:28.618618 | 
2026-03-09 14:03:28.618849 | TASK [controller : Write CA certificate to file]
2026-03-09 14:03:29.366326 | hotstack -> 38.102.83.163 | Output suppressed because no_log was given
2026-03-09 14:03:29.380497 | 
2026-03-09 14:03:29.380637 | TASK [controller : Write Ironic nodes YAML]
2026-03-09 14:03:30.145505 | hotstack -> 38.102.83.163 | changed
2026-03-09 14:03:30.175537 | 
2026-03-09 14:03:30.175840 | TASK [Install Openshift]
2026-03-09 14:03:30.203537 | hotstack | Output suppressed because no_log was given
2026-03-09 14:03:30.256141 | 
2026-03-09 14:03:30.256264 | TASK [ocp_agent_installer : Assert config is defined]
2026-03-09 14:03:30.331721 | hotstack -> 38.102.83.163 | ok: All assertions passed
2026-03-09 14:03:30.347996 | 
2026-03-09 14:03:30.348168 | TASK [ocp_agent_installer : Install using agent installer]
2026-03-09 14:03:30.402791 | hotstack | ok
2026-03-09 14:03:30.425108 | hotstack | included: /var/lib/zuul/builds/5ad27496488549c98262fbfcd2662052/trusted/project_3/github.com/openstack-k8s-operators/hotstack/roles/ocp_agent_installer/tasks/_install.yml
2026-03-09 14:03:30.435067 | 
2026-03-09 14:03:30.435190 | TASK [ocp_agent_installer : Assert config is defined]
2026-03-09 14:03:30.510642 | hotstack -> 38.102.83.163 | ok: All assertions passed
2026-03-09 14:03:30.518804 | 
2026-03-09 14:03:30.518909 | LOOP [ocp_agent_installer : Ensure directory exists]
2026-03-09 14:03:30.847783 | hotstack -> 38.102.83.163 | ok: "/home/zuul/bin"
2026-03-09 14:03:30.848160 | hotstack -> 38.102.83.163 | changed: All items complete
2026-03-09 14:03:30.848207 | 
2026-03-09 14:03:31.110772 | hotstack -> 38.102.83.163 | changed: "/home/zuul/.kube"
2026-03-09 14:03:31.400108 | hotstack -> 38.102.83.163 | changed: "/home/zuul/ocp-cluster"
2026-03-09 14:03:31.743188 | hotstack -> 38.102.83.163 | changed: "/home/zuul/ocp-cluster/openshift"
2026-03-09 14:03:32.012441 | hotstack -> 38.102.83.163 | changed: "/home/zuul/agent-installer"
2026-03-09 14:03:32.266004 | hotstack -> 38.102.83.163 | changed: "/home/zuul/cluster-custom-config/"
2026-03-09 14:03:32.560394 | hotstack -> 38.102.83.163 | changed: "/home/zuul/cluster-custom-config//butane"
2026-03-09 14:03:32.827583 | hotstack -> 38.102.83.163 | changed: "/home/zuul/cluster-custom-config//machine-configs"
2026-03-09 14:03:33.083691 | hotstack -> 38.102.83.163 | changed: "/home/zuul/cluster-custom-config//config-assets"
2026-03-09 14:03:33.094935 | 
2026-03-09 14:03:33.095049 | TASK [ocp_agent_installer : Install package requirements for agent installer]
2026-03-09 14:03:34.428023 | hotstack -> 38.102.83.163 | ok: Nothing to do
2026-03-09 14:03:34.442274 | 
2026-03-09 14:03:34.442435 | TASK [ocp_agent_installer : Run tasks/install_client.yml]
2026-03-09 14:03:34.474545 | hotstack | ok
2026-03-09 14:03:34.482975 | hotstack | included: /var/lib/zuul/builds/5ad27496488549c98262fbfcd2662052/trusted/project_3/github.com/openstack-k8s-operators/hotstack/roles/ocp_agent_installer/tasks/install_client.yml
2026-03-09 14:03:34.494297 | 
2026-03-09 14:03:34.494371 | TASK [ocp_agent_installer : Download the client]
2026-03-09 14:03:37.910905 | hotstack -> 38.102.83.163 | changed: OK (73357262 bytes)
2026-03-09 14:03:37.919621 | 
2026-03-09 14:03:37.919707 | TASK [ocp_agent_installer : Extract client to /home/zuul/bin]
2026-03-09 14:03:45.077877 | hotstack -> 38.102.83.163 | changed
2026-03-09 14:03:45.097824 | 
2026-03-09 14:03:45.097998 | TASK [ocp_agent_installer : Configure bash completion]
2026-03-09 14:03:46.135069 | hotstack -> 38.102.83.163 | changed
2026-03-09 14:03:46.148570 | 
2026-03-09 14:03:46.148811 | TASK [ocp_agent_installer : Run tasks/install_installer.yml]
2026-03-09 14:03:46.188510 | hotstack | ok
2026-03-09 14:03:46.205795 | hotstack | included: /var/lib/zuul/builds/5ad27496488549c98262fbfcd2662052/trusted/project_3/github.com/openstack-k8s-operators/hotstack/roles/ocp_agent_installer/tasks/install_installer.yml
2026-03-09 14:03:46.227760 | 
2026-03-09 14:03:46.227883 | TASK [ocp_agent_installer : Download the installer]
2026-03-09 14:03:56.934030 | hotstack -> 38.102.83.163 | changed: OK (437747386 bytes)
2026-03-09 14:03:56.958974 | 
2026-03-09 14:03:56.959105 | TASK [ocp_agent_installer : Extract installer to /home/zuul/bin]
2026-03-09 14:04:10.499931 | hotstack -> 38.102.83.163 | changed
2026-03-09 14:04:10.514364 | 
2026-03-09 14:04:10.514531 | TASK [ocp_agent_installer : Write OCP install-config.yaml]
2026-03-09 14:04:11.530550 | hotstack -> 38.102.83.163 | changed
2026-03-09 14:04:11.545041 | 
2026-03-09 14:04:11.545202 | TASK [ocp_agent_installer : Replace pull secret in install-config.yaml]
2026-03-09 14:04:12.074082 | hotstack -> 38.102.83.163 | Output suppressed because no_log was given
2026-03-09 14:04:12.087969 | 
2026-03-09 14:04:12.088119 | TASK [ocp_agent_installer : Write OCP agent-config.yaml]
2026-03-09 14:04:12.779079 | hotstack -> 38.102.83.163 | changed
2026-03-09 14:04:12.792215 | 
2026-03-09 14:04:12.792352 | TASK [ocp_agent_installer : Include tasks to generate machine configs]
2026-03-09 14:04:12.834004 | hotstack | ok
2026-03-09 14:04:12.855437 | hotstack | included: /var/lib/zuul/builds/5ad27496488549c98262fbfcd2662052/trusted/project_3/github.com/openstack-k8s-operators/hotstack/roles/ocp_agent_installer/tasks/machine_configs.yml
2026-03-09 14:04:12.882861 | 
2026-03-09 14:04:12.883009 | LOOP [ocp_agent_installer : Template butane config for net.ifnames]
2026-03-09 14:04:13.668072 | hotstack -> 38.102.83.163 | changed: "master"
2026-03-09 14:04:13.689306 | 
2026-03-09 14:04:13.689514 | LOOP [ocp_agent_installer : Generate MachineConfig for iscsi]
2026-03-09 14:04:14.156037 | hotstack -> 38.102.83.163 | changed
2026-03-09 14:04:14.156695 | hotstack -> 38.102.83.163 | changed: All items complete
2026-03-09 14:04:14.156810 | 
2026-03-09 14:04:14.182993 | 
2026-03-09 14:04:14.183190 | LOOP [ocp_agent_installer : Template butane config for iscsi]
2026-03-09 14:04:14.996220 | hotstack -> 38.102.83.163 | changed: "master"
2026-03-09 14:04:15.008430 | 
2026-03-09 14:04:15.008605 | LOOP [ocp_agent_installer : Generate MachineConfig for iscsi]
2026-03-09 14:04:15.361042 | hotstack -> 38.102.83.163 | changed
2026-03-09 14:04:15.361449 | hotstack -> 38.102.83.163 | changed: All items complete
2026-03-09 14:04:15.361491 | 
2026-03-09 14:04:15.371828 | 
2026-03-09 14:04:15.371980 | LOOP [ocp_agent_installer : Template butane config for multipath]
2026-03-09 14:04:16.089830 | hotstack -> 38.102.83.163 | changed: "master"
2026-03-09 14:04:16.108596 | 
2026-03-09 14:04:16.108857 | LOOP [ocp_agent_installer : Generate MachineConfig for multipath]
2026-03-09 14:04:16.472126 | hotstack -> 38.102.83.163 | changed
2026-03-09 14:04:16.472450 | hotstack -> 38.102.83.163 | changed: All items complete
2026-03-09 14:04:16.472492 | 
2026-03-09 14:04:16.483678 | 
2026-03-09 14:04:16.483814 | LOOP [ocp_agent_installer : Template butane config for LVM cinder-volumes]
2026-03-09 14:04:17.230286 | hotstack -> 38.102.83.163 | changed: "master"
2026-03-09 14:04:17.253206 | 
2026-03-09 14:04:17.253458 | LOOP [ocp_agent_installer : Generate MachineConfig for multipath]
2026-03-09 14:04:17.687434 | hotstack -> 38.102.83.163 | changed
2026-03-09 14:04:17.687871 | hotstack -> 38.102.83.163 | changed: All items complete
2026-03-09 14:04:17.688130 | 
2026-03-09 14:04:17.706658 | 
2026-03-09 14:04:17.706916 | TASK [ocp_agent_installer : Include tasks to generate config assets]
2026-03-09 14:04:17.744257 | hotstack | ok
2026-03-09 14:04:17.758968 | hotstack | included: /var/lib/zuul/builds/5ad27496488549c98262fbfcd2662052/trusted/project_3/github.com/openstack-k8s-operators/hotstack/roles/ocp_agent_installer/tasks/config_assets.yml
2026-03-09 14:04:17.786302 | 
2026-03-09 14:04:17.786441 | TASK [ocp_agent_installer : Template ovn-k8s customization]
2026-03-09 14:04:18.469379 | hotstack -> 38.102.83.163 | changed
2026-03-09 14:04:18.484751 | 
2026-03-09 14:04:18.484972 | TASK [ocp_agent_installer : Copy Etcd customization]
2026-03-09 14:04:19.207122 | hotstack -> 38.102.83.163 | changed
2026-03-09 14:04:19.216872 | 
2026-03-09 14:04:19.216968 | TASK [ocp_agent_installer : Template ImageContentSourcePolicy customization]
2026-03-09 14:04:19.274547 | hotstack | skipping: Conditional result was False
2026-03-09 14:04:19.288474 | 
2026-03-09 14:04:19.288694 | TASK [ocp_agent_installer : Initialize _ocp_additional_trusted_ca_map fact]
2026-03-09 14:04:19.368957 | hotstack | skipping: Conditional result was False
2026-03-09 14:04:19.383632 | 
2026-03-09 14:04:19.383878 | LOOP [ocp_agent_installer : Append to _ocp_additional_trusted_ca_map fact]
2026-03-09 14:04:19.429941 | hotstack | skipping: Conditional result was False
2026-03-09 14:04:19.448290 | 
2026-03-09 14:04:19.448536 | TASK [ocp_agent_installer : Template additional CA config map]
2026-03-09 14:04:19.501654 | hotstack | skipping: Conditional result was False
2026-03-09 14:04:19.517151 | 
2026-03-09 14:04:19.517385 | TASK [ocp_agent_installer : Template Image config (additionalTrustedCA and/or insecureRegistries)]
2026-03-09 14:04:19.607695 | hotstack | skipping: Conditional result was False
2026-03-09 14:04:19.621471 | 
2026-03-09 14:04:19.621672 | TASK [ocp_agent_installer : Copy machine_configs to manifests dir]
2026-03-09 14:04:20.054444 | hotstack -> 38.102.83.163 | changed
2026-03-09 14:04:20.061592 | 
2026-03-09 14:04:20.061773 | TASK [ocp_agent_installer : Copy config assets to manifests dir]
2026-03-09 14:04:20.475664 | hotstack -> 38.102.83.163 | changed
2026-03-09 14:04:20.490879 | 
2026-03-09 14:04:20.491040 | TASK [ocp_agent_installer : Run tasks/pxe_assets.yml]
2026-03-09 14:04:20.533539 | hotstack | ok
2026-03-09 14:04:20.553518 | hotstack | included: /var/lib/zuul/builds/5ad27496488549c98262fbfcd2662052/trusted/project_3/github.com/openstack-k8s-operators/hotstack/roles/ocp_agent_installer/tasks/pxe_assets.yml
2026-03-09 14:04:20.589211 | 
2026-03-09 14:04:20.589410 | TASK [ocp_agent_installer : Create the boot-artifacts directory]
2026-03-09 14:04:20.993864 | hotstack -> 38.102.83.163 | changed
2026-03-09 14:04:21.011442 | 
2026-03-09 14:04:21.011675 | TASK [ocp_agent_installer : Create PXE assets]
2026-03-09 10:04:22.831524 | controller-0 | level=warning msg=Compute[0].Hyperthreading: Disabled is ignored
2026-03-09 10:04:23.435332 | controller-0 | level=warning msg=ControlPlane.Hyperthreading: Disabled is ignored
2026-03-09 10:04:23.435381 | controller-0 | level=info msg=Configuration has 1 master replicas and 0 worker replicas
2026-03-09 10:04:23.435396 | controller-0 | level=info msg=The rendezvous host IP (node0 IP) is 192.168.32.10
2026-03-09 10:04:24.163506 | controller-0 | level=info msg=Extracting base ISO from release payload
2026-03-09 10:04:56.992532 | controller-0 | level=info msg=Base ISO obtained from release and cached at [/home/zuul/.cache/agent/image_cache/coreos-x86_64.iso]
2026-03-09 10:05:53.553939 | controller-0 | level=info msg=Consuming Install Config from target directory
2026-03-09 10:05:53.567181 | controller-0 | level=info msg=Consuming Extra Manifests from target directory
2026-03-09 10:05:55.260741 | controller-0 | level=info msg=Consuming Agent Config from target directory
2026-03-09 10:05:55.261311 | controller-0 | level=info msg=Created iPXE script agent.x86_64.ipxe in boot-artifacts directory
2026-03-09 10:05:55.261390 | controller-0 | level=info msg=PXE boot artifacts created in: boot-artifacts
2026-03-09 10:05:55.261437 | controller-0 | level=info msg=Kernel parameters for PXE boot: rw ignition.firstboot ignition.platform.id=metal
2026-03-09 14:05:55.647199 | hotstack -> 38.102.83.163 | changed
2026-03-09 14:05:55.662140 | 
2026-03-09 14:05:55.662310 | TASK [ocp_agent_installer : Set serial console in ipxe]
2026-03-09 14:05:56.275333 | hotstack -> 38.102.83.163 | changed: line replaced
2026-03-09 14:05:56.285877 | 
2026-03-09 14:05:56.286041 | TASK [ocp_agent_installer : Disable net.ifnames]
2026-03-09 14:05:56.849334 | hotstack -> 38.102.83.163 | changed: line replaced
2026-03-09 14:05:56.863567 | 
2026-03-09 14:05:56.863722 | LOOP [ocp_agent_installer : Copy boot-artifacts to the web server - (ocp_agent_installer_boot_artifacts_dir)]
2026-03-09 14:05:57.737370 | hotstack -> 38.102.83.163 | changed: "agent.x86_64-vmlinuz"
2026-03-09 14:05:57.737718 | 
2026-03-09 14:05:59.004941 | hotstack -> 38.102.83.163 | changed: "agent.x86_64-initrd.img"
2026-03-09 14:06:47.229973 | hotstack -> 38.102.83.163 | changed: "agent.x86_64-rootfs.img"
2026-03-09 14:06:47.668612 | hotstack -> 38.102.83.163 | changed: "agent.x86_64.ipxe"
2026-03-09 14:06:47.686272 | 
2026-03-09 14:06:47.686396 | TASK [ocp_agent_installer : Run tasks/iso_assets.yml]
2026-03-09 14:06:47.723542 | hotstack | skipping: Conditional result was False
2026-03-09 14:06:47.739316 | 
2026-03-09 14:06:47.739625 | TASK [ocp_agent_installer : Copy auth/kubeconfig to ~/.kube/config]
2026-03-09 14:06:48.273536 | hotstack -> 38.102.83.163 | changed
2026-03-09 14:06:48.291906 | 
2026-03-09 14:06:48.292095 | TASK [ocp_agent_installer : Wait for bootstrap-complete]
2026-03-09 10:07:21.911627 | controller-0 | level=info msg=Waiting for cluster install to initialize. Sleeping for 30 seconds
2026-03-09 10:07:53.916145 | controller-0 | level=info msg=Waiting for cluster install to initialize. Sleeping for 30 seconds
2026-03-09 10:08:25.920371 | controller-0 | level=info msg=Waiting for cluster install to initialize. Sleeping for 30 seconds
2026-03-09 10:09:00.984574 | controller-0 | level=info msg=Cannot access Rendezvous Host. There may be a network configuration problem, check console for additional info
2026-03-09 10:09:32.989900 | controller-0 | level=info msg=Cannot access Rendezvous Host. There may be a network configuration problem, check console for additional info
2026-03-09 10:10:05.014087 | controller-0 | level=info msg=Cluster is not ready for install. Check validations
2026-03-09 10:10:05.018136 | controller-0 | level=info msg=Updated image information (Image type is "full-iso", SSH public key is set)
2026-03-09 10:10:07.031886 | controller-0 | level=info msg=Host 546b38de-51d5-4245-8cc0-49a6ad501489: Successfully registered
2026-03-09 10:10:09.044610 | controller-0 | level=warning msg=Cluster validation: The cluster has hosts that are not ready to install.
2026-03-09 10:10:15.078092 | controller-0 | level=warning msg=Host master-0 validation: Host couldn't synchronize with any NTP server
2026-03-09 10:10:15.080398 | controller-0 | level=warning msg=Host master-0: updated status from discovering to insufficient (Host cannot be installed due to following failing validation(s): Host couldn't synchronize with any NTP server)
2026-03-09 10:11:09.432302 | controller-0 | level=info msg=Host master-0 validation: Host NTP is synced
2026-03-09 10:11:09.433938 | controller-0 | level=info msg=Host master-0: updated status from insufficient to known (Host is ready to be installed)
2026-03-09 10:11:17.484634 | controller-0 | level=info msg=Cluster is ready for install
2026-03-09 10:11:17.485072 | controller-0 | level=info msg=Cluster validation: All hosts in the cluster are ready to install.
2026-03-09 10:11:23.529920 | controller-0 | level=info msg=Preparing cluster for installation
2026-03-09 10:11:25.544532 | controller-0 | level=info msg=Host master-0: updated status from known to preparing-for-installation (Host finished successfully to prepare for installation)
2026-03-09 10:12:22.199984 | controller-0 | level=info msg=Host master-0: New image status quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4c60342a9b9efba628155b01cd76cba0b9d54616013fcdf044fafe7700b09d2c. result: success. time: 2.03 seconds; size: 453.71 Megabytes; download rate: 234.20 MBps
2026-03-09 10:12:30.253691 | controller-0 | level=info msg=Host master-0: updated status from preparing-for-installation to preparing-successful (Host finished successfully to prepare for installation)
2026-03-09 10:12:38.309351 | controller-0 | level=info msg=Cluster installation in progress
2026-03-09 10:12:46.371671 | controller-0 | level=info msg=Host master-0: updated status from preparing-successful to installing (Installation is in progress)
2026-03-09 10:13:08.547171 | controller-0 | level=info msg=Host: master-0, reached installation stage Installing: bootstrap
2026-03-09 10:13:14.596579 | controller-0 | level=info msg=Host: master-0, reached installation stage Waiting for bootkube
2026-03-09 10:14:29.511411 | controller-0 | level=info msg=Bootstrap Kube API Initialized
2026-03-09 10:17:04.949558 | controller-0 | level=info msg=Host: master-0, reached installation stage Writing image to disk
2026-03-09 10:17:06.963724 | controller-0 | level=info msg=Host: master-0, reached installation stage Writing image to disk: 5%
2026-03-09 10:17:08.977340 | controller-0 | level=info msg=Host: master-0, reached installation stage Writing image to disk: 14%
2026-03-09 10:17:10.990629 | controller-0 | level=info msg=Host: master-0, reached installation stage Writing image to disk: 19%
2026-03-09 10:17:13.004601 | controller-0 | level=info msg=Host: master-0, reached installation stage Writing image to disk: 24%
2026-03-09 10:17:19.057328 | controller-0 | level=info msg=Host: master-0, reached installation stage Writing image to disk: 30%
2026-03-09 10:17:21.069355 | controller-0 | level=info msg=Host: master-0, reached installation stage Writing image to disk: 36%
2026-03-09 10:17:27.115760 | controller-0 | level=info msg=Host: master-0, reached installation stage Writing image to disk: 42%
2026-03-09 10:17:31.151006 | controller-0 | level=info msg=Host: master-0, reached installation stage Writing image to disk: 47%
2026-03-09 10:17:37.192723 | controller-0 | level=info msg=Host: master-0, reached installation stage Writing image to disk: 52%
2026-03-09 10:17:41.227258 | controller-0 | level=info msg=Host: master-0, reached installation stage Writing image to disk: 58%
2026-03-09 10:17:45.263527 | controller-0 | level=info msg=Host: master-0, reached installation stage Writing image to disk: 63%
2026-03-09 10:17:49.394274 | controller-0 | level=info msg=Host: master-0, reached installation stage Writing image to disk: 68%
2026-03-09 10:17:55.448928 | controller-0 | level=info msg=Host: master-0, reached installation stage Writing image to disk: 73%
2026-03-09 10:17:59.483037 | controller-0 | level=info msg=Host: master-0, reached installation stage Writing image to disk: 79%
2026-03-09 10:18:05.683922 | controller-0 | level=info msg=Host: master-0, reached installation stage Writing image to disk: 84%
2026-03-09 10:18:11.915417 | controller-0 | level=info msg=Host: master-0, reached installation stage Writing image to disk: 89%
2026-03-09 10:18:13.930686 | controller-0 | level=info msg=Host: master-0, reached installation stage Writing image to disk: 94%
2026-03-09 10:18:17.971629 | controller-0 | level=info msg=Host: master-0, reached installation stage Writing image to disk: 100%
2026-03-09 10:20:17.759308 | controller-0 | level=info msg=Bootstrap configMap status is complete
2026-03-09 10:20:19.759651 | controller-0 | level=info msg=Bootstrap is complete
2026-03-09 10:20:19.759821 | controller-0 | level=info msg=cluster bootstrap is complete
2026-03-09 14:20:20.012607 | hotstack -> 38.102.83.163 | changed
2026-03-09 14:20:20.029155 | 
2026-03-09 14:20:20.029343 | TASK [ocp_agent_installer : Wait for install-complete]
2026-03-09 10:20:21.014033 | controller-0 | level=info msg=Bootstrap Kube API Initialized
2026-03-09 10:20:21.016894 | controller-0 | level=info msg=Bootstrap configMap status is complete
2026-03-09 10:20:23.018474 | controller-0 | level=info msg=Bootstrap is complete
2026-03-09 10:20:23.019187 | controller-0 | level=info msg=cluster bootstrap is complete
2026-03-09 10:49:46.732904 | controller-0 | level=info msg=Cluster is installed
2026-03-09 10:49:46.735775 | controller-0 | level=info msg=Install complete!
2026-03-09 10:49:46.735864 | controller-0 | level=info msg=To access the cluster as the system:admin user when using 'oc', run
2026-03-09 10:49:46.735900 | controller-0 | level=info msg=    export KUBECONFIG=/home/zuul/ocp-cluster/auth/kubeconfig
2026-03-09 10:49:46.735946 | controller-0 | level=info msg=Access the OpenShift web-console here: https://console-openshift-console.apps.sno.openstack.lab
2026-03-09 10:49:46.735993 | controller-0 | level=info msg=Login to the console with user: "kubeadmin", and password: "teckt-jdtux-k7Mui-zcFcI"
2026-03-09 14:49:46.892839 | hotstack -> 38.102.83.163 | changed
2026-03-09 14:49:46.905667 | 
2026-03-09 14:49:46.905876 | TASK [ocp_agent_installer : Add ingress certificate to CA trust]
2026-03-09 14:49:46.965758 | hotstack | ok
2026-03-09 14:49:46.985446 | hotstack | included: /var/lib/zuul/builds/5ad27496488549c98262fbfcd2662052/trusted/project_3/github.com/openstack-k8s-operators/hotstack/roles/ocp_agent_installer/tasks/ingress_cert.yml
2026-03-09 14:49:47.039416 | 
2026-03-09 14:49:47.039795 | TASK [ocp_agent_installer : Extract ingress CA cert]
2026-03-09 14:49:48.567449 | hotstack -> 38.102.83.163 | Output suppressed because no_log was given
2026-03-09 14:49:48.583502 | 
2026-03-09 14:49:48.583761 | TASK [ocp_agent_installer : Write ingress cert to ca-trust]
2026-03-09 14:49:49.893486 | hotstack -> 38.102.83.163 | changed
2026-03-09 14:49:49.913478 | 
2026-03-09 14:49:49.913666 | TASK [ocp_agent_installer : Update CA trust]
2026-03-09 14:49:52.954813 | hotstack -> 38.102.83.163 | changed
2026-03-09 14:49:52.971996 | 
2026-03-09 14:49:52.972164 | TASK [ocp_agent_installer : Wait for 25 hours to prepare for snapshot]
2026-03-09 14:49:53.050681 | hotstack | skipping: Conditional result was False
2026-03-09 14:49:53.064558 | 
2026-03-09 14:49:53.064754 | TASK [ocp_agent_installer : Prepare for Snapshot - and shutdown]
2026-03-09 14:49:53.605028 | hotstack | skipping: Conditional result was False
2026-03-09 14:49:53.611225 | 
2026-03-09 14:49:53.611298 | TASK [ocp_agent_installer : Wait for cluster, uncordon, wait for cluster, wait for route API version ...]
2026-03-09 14:49:54.146213 | hotstack | skipping: Conditional result was False
2026-03-09 14:49:54.175334 | 
2026-03-09 14:49:54.175555 | TASK [Set up RedFish Virtual bmc]
2026-03-09 14:49:54.267621 | hotstack | Output suppressed because no_log was given
2026-03-09 14:49:54.356544 | 
2026-03-09 14:49:54.356706 | TASK [redfish_virtual_bmc : Create tempdir]
2026-03-09 14:49:54.967268 | hotstack | changed
2026-03-09 14:49:54.982462 | 
2026-03-09 14:49:54.982667 | TASK [redfish_virtual_bmc : Install package requirements for agent installer]
2026-03-09 14:49:56.738405 | hotstack | changed
2026-03-09 14:49:56.750063 | 
2026-03-09 14:49:56.750226 | TASK [redfish_virtual_bmc : Generate htpasswd]
2026-03-09 10:49:57.016187 | hotstack | admin:$2y$05$3R6zqoPSiIzlB8lTm96z4OSuRBwcjmAxRDppKChA59kuhZIVuGxUy
2026-03-09 10:49:57.016241 | hotstack |
2026-03-09 14:49:57.299543 | hotstack | changed
2026-03-09 14:49:57.312017 | 
2026-03-09 14:49:57.312174 | LOOP [redfish_virtual_bmc : Copy files to tempdir]
2026-03-09 14:49:57.932421 | hotstack | changed: "namespace.yaml"
2026-03-09 14:49:58.514663 | hotstack | changed: "deployment.yaml"
2026-03-09 14:49:59.131702 | hotstack | changed: "service.yaml"
2026-03-09 14:49:59.154457 | 
2026-03-09 14:49:59.154636 | TASK [redfish_virtual_bmc : Template the ConfigMap]
2026-03-09 14:49:59.834128 | hotstack | changed
2026-03-09 14:49:59.841577 | 
2026-03-09 14:49:59.841655 | TASK [redfish_virtual_bmc : Template the Route]
2026-03-09 14:50:00.628413 | hotstack | changed
2026-03-09 14:50:00.637927 | 
2026-03-09 14:50:00.638038 | TASK [redfish_virtual_bmc : Load automation vars]
2026-03-09 14:50:00.682626 | hotstack | ok
2026-03-09 14:50:00.689264 | 
2026-03-09 14:50:00.689340 | TASK [Run the hotloop]
2026-03-09 14:50:00.720271 | hotstack | ok
2026-03-09 14:50:00.748190 | 
2026-03-09 14:50:00.748281 | TASK [hotloop : Initialize retry metrics]
2026-03-09 14:50:00.782686 | hotstack -> 38.102.83.163 | ok
2026-03-09 14:50:00.789022 | 
2026-03-09 14:50:00.789090 | TASK [hotloop : Assert config is defined]
2026-03-09 14:50:00.874396 | hotstack -> 38.102.83.163 | ok: All assertions passed
2026-03-09 14:50:00.888661 | 
2026-03-09 14:50:00.888892 | TASK [hotloop : Load stages]
2026-03-09 14:50:01.560546 | hotstack -> 38.102.83.163 | changed
2026-03-09 14:50:01.577366 | 
2026-03-09 14:50:01.577567 | LOOP [hotloop : Ensure directory exists]
2026-03-09 14:50:02.157561 | hotstack -> 38.102.83.163 | changed: "/home/zuul/manifests/sushy_emulator_manifests"
2026-03-09 14:50:02.157866 | hotstack -> 38.102.83.163 | changed: All items complete
2026-03-09 14:50:02.157897 | 
2026-03-09 14:50:02.171425 | 
2026-03-09 14:50:02.171586 | TASK [hotloop : Create temporary hotloop work directory]
2026-03-09 14:50:02.718991 | hotstack -> 38.102.83.163 | changed
2026-03-09 14:50:02.733890 | 
2026-03-09 14:50:02.734049 | TASK [hotloop : Create temporary directory for templates]
2026-03-09 14:50:03.031497 | hotstack -> localhost | changed
2026-03-09 14:50:03.044903 | 
2026-03-09 14:50:03.045074 | TASK [hotloop : Remove target directory for clean sync (local connection)]
2026-03-09 14:50:03.104369 | hotstack | skipping: Conditional result was False
2026-03-09 14:50:03.118587 | 
2026-03-09 14:50:03.118767 | TASK [hotloop : Copy work files (local connection)]
2026-03-09 14:50:03.166997 | hotstack | skipping: Conditional result was False
2026-03-09 14:50:03.182931 | 
2026-03-09 14:50:03.183093 | TASK [hotloop : Sync work files (remote connection) - using synchronize push]
2026-03-09 14:50:04.049198 | hotstack -> 38.102.83.210 | changed:
2026-03-09 14:50:04.049526 | hotstack -> 38.102.83.210 | .d..t...... ./
2026-03-09 14:50:04.049583 | hotstack -> 38.102.83.210 | <f+++++++++ config_map.yaml
2026-03-09 14:50:04.049624 | hotstack -> 38.102.83.210 | <f+++++++++ deployment.yaml
2026-03-09 14:50:04.049661 | hotstack -> 38.102.83.210 | <f+++++++++ namespace.yaml
2026-03-09 14:50:04.049697 | hotstack -> 38.102.83.210 | <f+++++++++ route.yaml
2026-03-09 14:50:04.049766 | hotstack -> 38.102.83.210 | <f+++++++++ service.yaml
2026-03-09 14:50:04.063969 | 
2026-03-09 14:50:04.064138 | LOOP [hotloop : Render all stage task files]
2026-03-09 14:50:04.580791 | hotstack -> localhost | ok:
2026-03-09 14:50:04.581654 | hotstack -> localhost | {
2026-03-09 14:50:04.581694 | hotstack -> localhost |   "manifest": "namespace.yaml",
2026-03-09 14:50:04.581714 | hotstack -> localhost |   "name": "Sushy Emulator : Namespace",
2026-03-09 14:50:04.581757 | hotstack -> localhost |   "wait_conditions": [
2026-03-09 14:50:04.581776 | hotstack -> localhost |     "oc wait namespaces sushy-emulator --for jsonpath='{.status.phase}'=Active --timeout=300s"
2026-03-09 14:50:04.581791 | hotstack -> localhost |   ]
2026-03-09 14:50:04.581806 | hotstack -> localhost | }
2026-03-09 14:50:05.096048 | hotstack -> localhost | ok:
2026-03-09 14:50:05.096147 | hotstack -> localhost | {
2026-03-09 14:50:05.096170 | hotstack -> localhost |   "name": "Sushy Emulator : Openstack clouds secret",
2026-03-09 14:50:05.096193 | hotstack -> localhost |   "shell": "oc create -n sushy-emulator secret generic os-client-config --save-config --dry-run=client --from-file=openstack-clouds-yaml=/home/zuul/.hotcloud/clouds.yaml --from-file=certificate-pem=/home/zuul/.hotcloud/cacert.pem --type=Opaque -o yaml | oc apply -f -"
2026-03-09 14:50:05.096212 | hotstack -> localhost | }
2026-03-09 14:50:05.525832 | hotstack -> localhost | ok:
2026-03-09 14:50:05.525931 | hotstack -> localhost | {
2026-03-09 14:50:05.525953 | hotstack -> localhost |   "manifest": "config_map.yaml",
2026-03-09 14:50:05.525971 | hotstack -> localhost |   "name": "Sushy Emulator : Config Map",
2026-03-09 14:50:05.525988 | hotstack -> localhost |   "wait_conditions": [
2026-03-09 14:50:05.526004 | hotstack -> localhost |     "oc wait -n sushy-emulator configmaps sushy-emulator-config --for jsonpath='{.metadata.name}'=sushy-emulator-config --timeout=30s"
2026-03-09 14:50:05.526021 | hotstack -> localhost |   ]
2026-03-09 14:50:05.526036 | hotstack -> localhost | }
2026-03-09 14:50:06.024474 | hotstack -> localhost | ok:
2026-03-09 14:50:06.024656 | hotstack -> localhost | {
2026-03-09 14:50:06.024765 | hotstack -> localhost |   "manifest": "deployment.yaml",
2026-03-09 14:50:06.024821 | hotstack -> localhost |   "name": "Sushy Emulator : Deployment",
2026-03-09 14:50:06.024858 | hotstack -> localhost |   "wait_conditions": [
2026-03-09 14:50:06.024892 | hotstack -> localhost |     "oc wait -n sushy-emulator pod -l app=sushy-emulator --for condition=Ready --timeout=300s"
2026-03-09 14:50:06.024926 | hotstack -> localhost |   ]
2026-03-09 14:50:06.024960 | hotstack -> localhost | }
2026-03-09 14:50:06.477866 | hotstack -> localhost | ok:
2026-03-09 14:50:06.477968 | hotstack -> localhost | {
2026-03-09 14:50:06.477990 | hotstack -> localhost |   "manifest": "service.yaml",
2026-03-09 14:50:06.478020 | hotstack -> localhost |   "name": "Sushy Emulator : Service",
2026-03-09 14:50:06.478037 | hotstack -> localhost |   "wait_conditions": [
2026-03-09 14:50:06.478054 | hotstack -> localhost |     "oc wait -n sushy-emulator service/sushy-emulator-service --for jsonpath='{.status.loadBalancer}' --timeout=30s"
2026-03-09 14:50:06.478070 | hotstack -> localhost |   ]
2026-03-09 14:50:06.478086 | hotstack -> localhost | }
2026-03-09 14:50:06.937513 | hotstack -> localhost | ok:
2026-03-09 14:50:06.937625 | hotstack -> localhost | {
2026-03-09 14:50:06.937649 | hotstack -> localhost |   "name": "Sushy Emulator : Service :: Wait for Endpoints",
2026-03-09 14:50:06.937669 | hotstack -> localhost |   "wait_conditions": [
2026-03-09 14:50:06.937687 | hotstack -> localhost |     "oc wait -n sushy-emulator endpoints/sushy-emulator-service --for jsonpath='{.subsets[*].addresses[*].ip}' --timeout=120s"
2026-03-09 14:50:06.937705 | hotstack -> localhost |   ]
2026-03-09 14:50:06.937721 | hotstack -> localhost | }
2026-03-09 14:50:07.468817 | hotstack -> localhost | ok:
2026-03-09 14:50:07.468946 | hotstack -> localhost | {
2026-03-09 14:50:07.468982 | hotstack -> localhost |   "manifest": "route.yaml",
2026-03-09 14:50:07.469006 | hotstack -> localhost |   "name": "Sushy Emulator : Route",
2026-03-09 14:50:07.469030 | hotstack -> localhost |   "wait_conditions": [
2026-03-09 14:50:07.469051 | hotstack -> localhost |     "oc wait -n sushy-emulator routes/sushy-emulator-route --for jsonpath='{.status.ingress[*].conditions[*].status}'=True --timeout=60s"
2026-03-09 14:50:07.469073 | hotstack -> localhost |   ]
2026-03-09 14:50:07.469103 | hotstack -> localhost | }
2026-03-09 14:50:07.496284 | 
2026-03-09 14:50:07.496474 | LOOP [hotloop : Execute automation stages]
2026-03-09 14:50:07.530966 | hotstack | ok:
2026-03-09 14:50:07.531207 | hotstack | {
2026-03-09 14:50:07.531233 | hotstack |   "manifest": "namespace.yaml",
2026-03-09 14:50:07.531252 | hotstack |   "name": "Sushy Emulator : Namespace",
2026-03-09 14:50:07.531268 | hotstack |   "wait_conditions": [
2026-03-09 14:50:07.531284 | hotstack |     "oc wait namespaces sushy-emulator --for jsonpath='{.status.phase}'=Active --timeout=300s"
2026-03-09 14:50:07.531299 | hotstack |   ]
2026-03-09 14:50:07.531314 | hotstack | }
2026-03-09 14:50:07.531342 | hotstack | ok: All items complete
2026-03-09 14:50:07.531360 | 
2026-03-09 14:50:07.547435 | hotstack | ok:
2026-03-09 14:50:07.547751 | hotstack | {
2026-03-09 14:50:07.547783 | hotstack |   "name": "Sushy Emulator : Openstack clouds secret",
2026-03-09 14:50:07.547808 | hotstack |   "shell": "oc create -n sushy-emulator secret generic os-client-config --save-config --dry-run=client --from-file=openstack-clouds-yaml=/home/zuul/.hotcloud/clouds.yaml --from-file=certificate-pem=/home/zuul/.hotcloud/cacert.pem --type=Opaque -o yaml | oc apply -f -"
2026-03-09 14:50:07.547845 | hotstack | }
2026-03-09 14:50:07.555756 | hotstack | ok:
2026-03-09 14:50:07.555860 | hotstack | {
2026-03-09 14:50:07.555882 | hotstack |   "manifest": "config_map.yaml",
2026-03-09 14:50:07.555901 | hotstack |   "name": "Sushy Emulator : Config Map",
2026-03-09 14:50:07.555919 | hotstack |   "wait_conditions": [
2026-03-09 14:50:07.555936 | hotstack |     "oc wait -n sushy-emulator configmaps sushy-emulator-config --for jsonpath='{.metadata.name}'=sushy-emulator-config --timeout=30s"
2026-03-09 14:50:07.555952 | hotstack |   ]
2026-03-09 14:50:07.555968 | hotstack | }
2026-03-09 14:50:07.562330 | hotstack | ok:
2026-03-09 14:50:07.562409 | hotstack | {
2026-03-09 14:50:07.562432 | hotstack |   "manifest": "deployment.yaml",
2026-03-09 14:50:07.562450 | hotstack |   "name": "Sushy Emulator : Deployment",
2026-03-09 14:50:07.562468 | hotstack |   "wait_conditions": [
2026-03-09 14:50:07.562485 | hotstack |     "oc wait -n sushy-emulator pod -l app=sushy-emulator --for condition=Ready --timeout=300s"
2026-03-09 14:50:07.562501 | hotstack |   ]
2026-03-09 14:50:07.562519 | hotstack | }
2026-03-09 14:50:07.570341 | hotstack | ok:
2026-03-09 14:50:07.570454 | hotstack | {
2026-03-09 14:50:07.570488 | hotstack |   "manifest": "service.yaml",
2026-03-09 14:50:07.570514 | hotstack |   "name": "Sushy Emulator : Service",
2026-03-09 14:50:07.570534 | hotstack |   "wait_conditions": [
2026-03-09 14:50:07.570550 | hotstack |     "oc wait -n sushy-emulator service/sushy-emulator-service --for jsonpath='{.status.loadBalancer}' --timeout=30s"
2026-03-09 14:50:07.570566 | hotstack |   ]
2026-03-09 14:50:07.570581 | hotstack | }
2026-03-09 14:50:07.576243 | hotstack | ok:
2026-03-09 14:50:07.576295 | hotstack | {
2026-03-09 14:50:07.576318 | hotstack |   "name": "Sushy Emulator : Service :: Wait for Endpoints",
2026-03-09 14:50:07.576336 | hotstack |   "wait_conditions": [
2026-03-09 14:50:07.576352 | hotstack |     "oc wait -n sushy-emulator endpoints/sushy-emulator-service --for jsonpath='{.subsets[*].addresses[*].ip}' --timeout=120s"
2026-03-09 14:50:07.576368 | hotstack |   ]
2026-03-09 14:50:07.576383 | hotstack | }
2026-03-09 14:50:07.581821 | hotstack | ok:
2026-03-09 14:50:07.581912 | hotstack | {
2026-03-09 14:50:07.581939 | hotstack |   "manifest": "route.yaml",
2026-03-09 14:50:07.581958 | hotstack |   "name": "Sushy Emulator : Route",
2026-03-09 14:50:07.581974 | hotstack |   "wait_conditions": [
2026-03-09 14:50:07.581993 | hotstack |     "oc wait -n sushy-emulator routes/sushy-emulator-route --for jsonpath='{.status.ingress[*].conditions[*].status}'=True --timeout=60s"
2026-03-09 14:50:07.582020 | hotstack |   ]
2026-03-09 14:50:07.582047 | hotstack | }
2026-03-09 14:50:07.607564 | hotstack | included: /var/lib/zuul/builds/5ad27496488549c98262fbfcd2662052/work/tmp/ansible.yi1whivxhotloop_templates/stage_1.yml
2026-03-09 14:50:07.608975 | hotstack | included: /var/lib/zuul/builds/5ad27496488549c98262fbfcd2662052/work/tmp/ansible.yi1whivxhotloop_templates/stage_2.yml
2026-03-09 14:50:07.610773 | hotstack | included: /var/lib/zuul/builds/5ad27496488549c98262fbfcd2662052/work/tmp/ansible.yi1whivxhotloop_templates/stage_3.yml
2026-03-09 14:50:07.612803 | hotstack | included: /var/lib/zuul/builds/5ad27496488549c98262fbfcd2662052/work/tmp/ansible.yi1whivxhotloop_templates/stage_4.yml
2026-03-09 14:50:07.614711 | hotstack | included: /var/lib/zuul/builds/5ad27496488549c98262fbfcd2662052/work/tmp/ansible.yi1whivxhotloop_templates/stage_5.yml
2026-03-09 14:50:07.616193 | hotstack | included: /var/lib/zuul/builds/5ad27496488549c98262fbfcd2662052/work/tmp/ansible.yi1whivxhotloop_templates/stage_6.yml
2026-03-09 14:50:07.618012 | hotstack | included: /var/lib/zuul/builds/5ad27496488549c98262fbfcd2662052/work/tmp/ansible.yi1whivxhotloop_templates/stage_7.yml
2026-03-09 14:50:07.626013 | 
2026-03-09 14:50:07.626081 | TASK [hotloop : Stage: Sushy Emulator : Namespace :: Apply static manifest]
2026-03-09 14:50:07.656129 | hotstack | ok
2026-03-09 14:50:07.667572 | hotstack | included: /var/lib/zuul/builds/5ad27496488549c98262fbfcd2662052/trusted/project_3/github.com/openstack-k8s-operators/hotstack/roles/hotloop/tasks/static_manifest.yml
2026-03-09 14:50:07.675635 | 
2026-03-09 14:50:07.675738 | TASK [hotloop : Stage: Sushy Emulator : Namespace :: Ensure manifest dir]
2026-03-09 14:50:08.262905 | hotstack -> 38.102.83.163 | ok
2026-03-09 14:50:08.278699 | 
2026-03-09 14:50:08.278899 | TASK [hotloop : Stage: Sushy Emulator : Namespace :: Check if manifest exists in synced work directory]
2026-03-09 14:50:08.667671 | hotstack -> 38.102.83.163 | ok
2026-03-09 14:50:08.686223 | 
2026-03-09 14:50:08.686399 | TASK [hotloop : Stage: Sushy Emulator : Namespace :: Copy manifest]
2026-03-09 14:50:09.145059 | hotstack -> 38.102.83.163 | changed
2026-03-09 14:50:09.160428 | 
2026-03-09 14:50:09.160587 | LOOP [hotloop : Stage: Sushy Emulator : Namespace :: Apply patches]
2026-03-09 14:50:09.215627 | 
2026-03-09 14:50:09.215965 | TASK [hotloop : Stage: Sushy Emulator : Namespace :: Apply static manifest]
2026-03-09 14:50:09.846234 | hotstack -> 38.102.83.163 | changed: Manifest file /home/zuul/manifests/sushy_emulator_manifests/namespace.yaml applied and saved as /home/zuul/manifests/sushy_emulator_manifests/namespace.yaml.applied with log at /home/zuul/manifests/sushy_emulator_manifests/namespace.yaml.applied.log
2026-03-09 14:50:09.854206 | 
2026-03-09 14:50:09.854321 | LOOP [hotloop : Stage: Sushy Emulator : Namespace :: Wait conditions]
2026-03-09 14:50:10.496812 | hotstack -> 38.102.83.163 | ok: "oc wait namespaces sushy-emulator --for jsonpath='{.status.phase}'=Active --timeout=300s"
2026-03-09 14:50:10.497477 | 
2026-03-09 14:50:10.514409 | 
2026-03-09 14:50:10.514537 | TASK [hotloop : Stage: Sushy Emulator : Openstack clouds secret :: Run shell]
2026-03-09 10:50:11.112801 | controller-0 | secret/os-client-config created
2026-03-09 14:50:11.550356 | hotstack -> 38.102.83.163 | changed
2026-03-09 14:50:11.564096 | 
2026-03-09 14:50:11.570033 | TASK [hotloop : Stage: Sushy Emulator : Config Map :: Apply static manifest]
2026-03-09 14:50:11.609416 | hotstack | ok
2026-03-09 14:50:11.621807 | hotstack | included: /var/lib/zuul/builds/5ad27496488549c98262fbfcd2662052/trusted/project_3/github.com/openstack-k8s-operators/hotstack/roles/hotloop/tasks/static_manifest.yml
2026-03-09 14:50:11.631388 | 
2026-03-09 14:50:11.631479 | TASK [hotloop : Stage: Sushy Emulator : Config Map :: Ensure manifest dir]
2026-03-09 14:50:12.171088 | hotstack -> 38.102.83.163 | ok
2026-03-09 14:50:12.189644 | 
2026-03-09 14:50:12.189877 | TASK [hotloop : Stage: Sushy Emulator : Config Map :: Check if manifest exists in synced work directory]
2026-03-09 14:50:12.815441 | hotstack -> 38.102.83.163 | ok
2026-03-09 14:50:12.837698 | 
2026-03-09 14:50:12.837925 | TASK [hotloop : Stage: Sushy Emulator : Config Map :: Copy manifest]
2026-03-09 14:50:13.613462 | hotstack -> 38.102.83.163 | changed
2026-03-09 14:50:13.641430 | 
2026-03-09 14:50:13.641570 | LOOP [hotloop : Stage: Sushy Emulator : Config Map :: Apply patches]
2026-03-09 14:50:13.705862 | 
2026-03-09 14:50:13.706108 | TASK [hotloop : Stage: Sushy Emulator : Config Map :: Apply static manifest]
2026-03-09 14:50:14.963706 | hotstack -> 38.102.83.163 | changed: Manifest file /home/zuul/manifests/sushy_emulator_manifests/config_map.yaml applied and saved as /home/zuul/manifests/sushy_emulator_manifests/config_map.yaml.applied with log at /home/zuul/manifests/sushy_emulator_manifests/config_map.yaml.applied.log
2026-03-09 14:50:14.981768 | 
2026-03-09 14:50:14.981965 | LOOP [hotloop : Stage: Sushy Emulator : Config Map :: Wait conditions]
2026-03-09 14:50:15.957945 | hotstack -> 38.102.83.163 | ok: "oc wait -n sushy-emulator configmaps sushy-emulator-config --for jsonpath='{.metadata.name}'=sushy-emulator-config --timeout=30s"
2026-03-09 14:50:15.958400 | 
2026-03-09 14:50:15.970324 | 
2026-03-09 14:50:15.970420 | TASK [hotloop : Stage: Sushy Emulator : Deployment :: Apply static manifest]
2026-03-09 14:50:15.999432 | hotstack | ok
2026-03-09 14:50:16.008160 | hotstack | included: /var/lib/zuul/builds/5ad27496488549c98262fbfcd2662052/trusted/project_3/github.com/openstack-k8s-operators/hotstack/roles/hotloop/tasks/static_manifest.yml
2026-03-09 14:50:16.015325 | 
2026-03-09 14:50:16.015391 | TASK [hotloop : Stage: Sushy Emulator : Deployment :: Ensure manifest dir]
2026-03-09 14:50:16.601787 | hotstack -> 38.102.83.163 | ok
2026-03-09 14:50:16.611044 | 
2026-03-09 14:50:16.611183 | TASK [hotloop : Stage: Sushy Emulator : Deployment :: Check if manifest exists in synced work directory]
2026-03-09 14:50:17.180942 | hotstack -> 38.102.83.163 | ok
2026-03-09 14:50:17.196575 | 
2026-03-09 14:50:17.196775 | TASK [hotloop : Stage: Sushy Emulator : Deployment :: Copy manifest]
2026-03-09 14:50:17.919267 | hotstack -> 38.102.83.163 | changed
2026-03-09 14:50:17.935925 | 
2026-03-09 14:50:17.936225 | LOOP [hotloop : Stage: Sushy Emulator : Deployment :: Apply patches]
2026-03-09 14:50:18.013285 | 
2026-03-09 14:50:18.013584 | TASK [hotloop : Stage: Sushy Emulator : Deployment :: Apply static manifest]
2026-03-09 14:50:18.962543 | hotstack -> 38.102.83.163 | changed: Manifest file /home/zuul/manifests/sushy_emulator_manifests/deployment.yaml applied and saved as /home/zuul/manifests/sushy_emulator_manifests/deployment.yaml.applied with log at /home/zuul/manifests/sushy_emulator_manifests/deployment.yaml.applied.log
2026-03-09 14:50:18.969820 | 
2026-03-09 14:50:18.969927 | LOOP [hotloop : Stage: Sushy Emulator : Deployment :: Wait conditions]
2026-03-09 14:51:22.603173 | hotstack -> 38.102.83.163 | ok: "oc wait -n sushy-emulator pod -l app=sushy-emulator --for condition=Ready --timeout=300s"
2026-03-09 14:51:22.603472 | 
2026-03-09 14:51:22.624582 | 
2026-03-09 14:51:22.624899 | TASK [hotloop : Stage: Sushy Emulator : Service :: Apply static manifest]
2026-03-09 14:51:22.667330 | hotstack | ok
2026-03-09 14:51:22.692844 | hotstack | included: /var/lib/zuul/builds/5ad27496488549c98262fbfcd2662052/trusted/project_3/github.com/openstack-k8s-operators/hotstack/roles/hotloop/tasks/static_manifest.yml
2026-03-09 14:51:22.706387 | 
2026-03-09 14:51:22.706534 | TASK [hotloop : Stage: Sushy Emulator : Service :: Ensure manifest dir]
2026-03-09 14:51:23.080416 | hotstack -> 38.102.83.163 | ok
2026-03-09 14:51:23.092502 | 
2026-03-09 14:51:23.092646 | TASK [hotloop : Stage: Sushy Emulator : Service :: Check if manifest exists in synced work directory]
2026-03-09 14:51:23.410620 | hotstack -> 38.102.83.163 | ok
2026-03-09 14:51:23.427813 | 
2026-03-09 14:51:23.428000 | TASK [hotloop : Stage: Sushy Emulator : Service :: Copy manifest]
2026-03-09 14:51:23.957594 | hotstack -> 38.102.83.163 | changed
2026-03-09 14:51:23.975625 | 
2026-03-09 14:51:23.975828 | LOOP [hotloop : Stage: Sushy Emulator : Service :: Apply patches]
2026-03-09 14:51:24.048653 | 
2026-03-09 14:51:24.049006 | TASK [hotloop : Stage: Sushy Emulator : Service :: Apply static manifest]
2026-03-09 14:51:24.548234 | hotstack -> 38.102.83.163 | changed: Manifest file /home/zuul/manifests/sushy_emulator_manifests/service.yaml applied and saved as /home/zuul/manifests/sushy_emulator_manifests/service.yaml.applied with log at /home/zuul/manifests/sushy_emulator_manifests/service.yaml.applied.log
2026-03-09 14:51:24.560206 | 
2026-03-09 14:51:24.560433 | LOOP [hotloop : Stage: Sushy Emulator : Service :: Wait conditions]
2026-03-09 14:51:25.129035 | hotstack -> 38.102.83.163 | ok: "oc wait -n sushy-emulator service/sushy-emulator-service --for jsonpath='{.status.loadBalancer}' --timeout=30s"
2026-03-09 14:51:25.129472 | 
2026-03-09 14:51:25.146950 | 
2026-03-09 14:51:25.147103 | LOOP [hotloop : Stage: Sushy Emulator : Service :: Wait for Endpoints :: Wait conditions]
2026-03-09 14:52:35.889903 | hotstack -> 38.102.83.163 | ERROR: Item: oc wait -n sushy-emulator endpoints/sushy-emulator-service --for jsonpath='{.subsets[*].addresses[*].ip}' --timeout=120s
2026-03-09 14:52:35.890471 | hotstack -> 38.102.83.163 | {
2026-03-09 14:52:35.890541 | hotstack -> 38.102.83.163 |   "ansible_loop_var": "wait_cmd",
2026-03-09 14:52:35.890584 | hotstack -> 38.102.83.163 |   "attempts": 3,
2026-03-09 14:52:35.890621 | hotstack -> 38.102.83.163 |   "elapsed_time": 70.4331705570221,
2026-03-09 14:52:35.890657 | hotstack -> 38.102.83.163 |   "msg": "Wait condition failed with non-retryable error after 3 attempts: oc wait -n sushy-emulator endpoints/sushy-emulator-service --for jsonpath='{.subsets[*].addresses[*].ip}' --timeout=120s",
2026-03-09 14:52:35.890696 | hotstack -> 38.102.83.163 |   "rc": 1,
2026-03-09 14:52:35.890758 | hotstack -> 38.102.83.163 |   "wait_cmd": "oc wait -n sushy-emulator endpoints/sushy-emulator-service --for jsonpath='{.subsets[*].addresses[*].ip}' --timeout=120s"
2026-03-09 14:52:35.890802 | hotstack -> 38.102.83.163 | }
failure
2026-03-09 14:52:35.916808 | 
2026-03-09 14:52:35.916939 | TASK [redfish_virtual_bmc : Clean up the tempdir]
2026-03-09 14:52:36.281639 | hotstack | changed
2026-03-09 14:52:36.284533 | 
2026-03-09 14:52:36.284626 | PLAY RECAP
2026-03-09 14:52:36.284868 | hotstack | ok: 136 changed: 79 unreachable: 0 failed: 1 skipped: 20 rescued: 0 ignored: 0
2026-03-09 14:52:36.284929 | 
2026-03-09 14:52:36.534525 | PRE-RUN END RESULT_NORMAL: [trusted : review.rdoproject.org/config/playbooks/hotstack/pre_stage2.yaml@master]
2026-03-09 14:52:36.537450 | POST-RUN START: [trusted : review.rdoproject.org/config/playbooks/hotstack/post.yaml@master]
2026-03-09 14:52:37.143313 | 
2026-03-09 14:52:37.143433 | PLAY [Collect hotstack logs]
2026-03-09 14:52:37.167141 | 
2026-03-09 14:52:37.167274 | TASK [Fetch stack outputs from remote to zuul executor]
2026-03-09 14:52:37.705444 | hotstack | Output suppressed because no_log was given
2026-03-09 14:52:37.718112 | 
2026-03-09 14:52:37.718340 | TASK [Load stack output vars with template resolution]
2026-03-09 14:52:37.767968 | hotstack | Output suppressed because no_log was given
2026-03-09 14:52:37.780454 | 
2026-03-09 14:52:37.780663 | TASK [Add Hotstack controller-0 to ansible inventory]
2026-03-09 14:52:37.830054 | hotstack | Output suppressed because no_log was given
2026-03-09 14:52:37.842355 | 
2026-03-09 14:52:37.842534 | TASK [Wait for Hotstack controller-0 to be ready]
2026-03-09 14:52:43.489406 | hotstack -> 38.102.83.163 | Output suppressed because no_log was given
2026-03-09 14:52:43.503025 | 
2026-03-09 14:52:43.503218 | TASK [Collect hotstack logs]
2026-03-09 14:52:43.533290 | hotstack | Output suppressed because no_log was given
2026-03-09 14:52:43.595747 | 
2026-03-09 14:52:43.595924 | TASK [hotlogs : Initialize failure tracking]
2026-03-09 14:52:43.653320 | hotstack | ok
2026-03-09 14:52:43.663163 | 
2026-03-09 14:52:43.663317 | TASK [hotlogs : Ensure logs directory]
2026-03-09 14:52:44.038184 | hotstack | changed
2026-03-09 14:52:44.050469 | 
2026-03-09 14:52:44.050700 | LOOP [hotlogs : Create hotlogs sub dirs]
2026-03-09 14:52:44.298274 | hotstack | changed:
2026-03-09 14:52:44.299035 | hotstack | {
2026-03-09 14:52:44.303362 | hotstack |   "dest_dir": "ocp_cluster",
2026-03-09 14:52:44.303499 | hotstack |   "src": "/home/zuul/ocp-cluster/.openshift_install.log"
2026-03-09 14:52:44.303538 | hotstack | }
2026-03-09 14:52:44.468098 | hotstack | ok:
2026-03-09 14:52:44.468196 | hotstack | {
2026-03-09 14:52:44.468219 | hotstack |   "dest_dir": "ocp_cluster",
2026-03-09 14:52:44.468238 | hotstack |   "src": "/home/zuul/cluster-custom-config/"
2026-03-09 14:52:44.468258 | hotstack | }
2026-03-09 14:52:44.652803 | hotstack | changed:
2026-03-09 14:52:44.653123 | hotstack | {
2026-03-09 14:52:44.653418 | hotstack |   "dest_dir": "data",
2026-03-09 14:52:44.653472 | hotstack |   "src": "/home/zuul/data/"
2026-03-09 14:52:44.653509 | hotstack | }
2026-03-09 14:52:44.841201 | hotstack | changed:
2026-03-09 14:52:44.841779 | hotstack | {
2026-03-09 14:52:44.841855 | hotstack |   "dest_dir": "manifests",
2026-03-09 14:52:44.841900 | hotstack |   "src": "/home/zuul/manifests/"
2026-03-09 14:52:44.841937 | hotstack | }
2026-03-09 14:52:45.020082 | hotstack | ok:
2026-03-09 14:52:45.020300 | hotstack | {
2026-03-09 14:52:45.020352 | hotstack |   "dest_dir": ".",
2026-03-09 14:52:45.020393 | hotstack |   "src": "/home/zuul/must-gather.tar.gz"
2026-03-09 14:52:45.020430 | hotstack | }
2026-03-09 14:52:45.201126 | hotstack | ok:
2026-03-09 14:52:45.201291 | hotstack | {
2026-03-09 14:52:45.201345 | hotstack |   "dest_dir": ".",
2026-03-09 14:52:45.201387 | hotstack |   "src": "/home/zuul/logs/tempest/"
2026-03-09 14:52:45.201425 | hotstack | }
2026-03-09 14:52:45.389135 | hotstack | changed:
2026-03-09 14:52:45.389301 | hotstack | {
2026-03-09 14:52:45.389368 | hotstack |   "dest_dir": "nova-console-recordings",
2026-03-09 14:52:45.389414 | hotstack |   "src": "/home/zuul/nova-console-recordings/"
2026-03-09 14:52:45.389454 | hotstack | }
2026-03-09 14:52:45.410412 | 
2026-03-09 14:52:45.410565 | TASK [hotlogs : Run must-gather operations]
2026-03-09 14:53:00.113821 | hotstack -> 38.102.83.163 | ERROR
2026-03-09 14:53:00.114042 | hotstack -> 38.102.83.163 | {
2026-03-09 14:53:00.114074 | hotstack -> 38.102.83.163 |   "msg": "Must-gather failed: Error from server (Forbidden): pods \"must-gather-\" is forbidden: error looking up service account openshift-must-gather-jsqzz/default: serviceaccount \"default\" not found\nError from server (Forbidden): pods \"must-gather-\" is forbidden: error looking up service account openshift-must-gather-jsqzz/default: serviceaccount \"default\" not found\n (Command: oc adm must-gather --image-stream=openshift/must-gather --image=quay.io/openstack-k8s-operators/openstack-must-gather --dest-dir=/home/zuul/must-gather --timeout=10m --host-network=false -- ADDITIONAL_NAMESPACES=sushy-emulator OPENSTACK_DATABASES=ALL SOS_EDPM=all SOS_DECOMPRESS=0 gather)",
2026-03-09 14:53:00.114099 | hotstack -> 38.102.83.163 |   "rc": 1
2026-03-09 14:53:00.114117 | hotstack -> 38.102.83.163 | }
2026-03-09 14:53:00.121174 | 
2026-03-09 14:53:00.121253 | TASK [hotlogs : Display must-gather failure]
2026-03-09 14:53:00.185832 | hotstack | ok:
2026-03-09 14:53:00.186036 | hotstack | Must-gather operations failed: Must-gather failed: Error from server (Forbidden): pods "must-gather-" is forbidden: error looking up service account openshift-must-gather-jsqzz/default: serviceaccount "default" not found
2026-03-09 14:53:00.186076 | hotstack | Error from server (Forbidden): pods "must-gather-" is forbidden: error looking up service account openshift-must-gather-jsqzz/default: serviceaccount "default" not found
2026-03-09 14:53:00.186108 | hotstack |  (Command: oc adm must-gather --image-stream=openshift/must-gather --image=quay.io/openstack-k8s-operators/openstack-must-gather --dest-dir=/home/zuul/must-gather --timeout=10m --host-network=false -- ADDITIONAL_NAMESPACES=sushy-emulator OPENSTACK_DATABASES=ALL SOS_EDPM=all SOS_DECOMPRESS=0 gather)
2026-03-09 14:53:00.195127 | 
2026-03-09 14:53:00.195220 | TASK [hotlogs : Track must-gather failure]
2026-03-09 14:53:00.249666 | hotstack | ok
2026-03-09 14:53:00.257597 | 
2026-03-09 14:53:00.257682 | LOOP [hotlogs : Collect tempest logs from test-operator PVCs]
2026-03-09 14:53:00.856938 | hotstack -> 38.102.83.163 | changed
2026-03-09 14:53:00.857353 | hotstack -> 38.102.83.163 | changed: All items complete
2026-03-09 14:53:00.857411 | 
2026-03-09 14:53:00.873297 | 
2026-03-09 14:53:00.873491 | LOOP [hotlogs : Display tempest log collection results]
2026-03-09 14:53:00.927084 | hotstack | ok: Namespace openstack: ['==========================================', 'Tempest Log Collection', '==========================================', 'Namespace: openstack', 'Logs directory: /home/zuul/logs/tempest', '==========================================', '', 'Searching for tempest log PVCs...', 'WARNING: No tempest PVCs found', "This is normal if tests haven't run yet or logs were already collected"]
2026-03-09 14:53:00.947073 | 
2026-03-09 14:53:00.947278 | LOOP [hotlogs : Track tempest log collection failures]
2026-03-09 14:53:00.966196 | hotstack | skipping: Conditional result was False
2026-03-09 14:53:00.966823 | hotstack | ok: All items complete
2026-03-09 14:53:00.966903 | 
2026-03-09 14:53:00.987984 | 
2026-03-09 14:53:00.988179 | TASK [hotlogs : Create nova console recordings directory in base_dir on controller]
2026-03-09 14:53:01.337439 | hotstack -> 38.102.83.163 | changed
2026-03-09 14:53:01.343355 | 
2026-03-09 14:53:01.343450 | TASK [hotlogs : Copy nova console recordings from NFS export]
2026-03-09 10:53:01.673912 | controller-0 | sending incremental file list
2026-03-09 10:53:01.675751 | controller-0 | ./
2026-03-09 10:53:01.717019 | controller-0 |
2026-03-09 10:53:01.717073 | controller-0 | sent 66 bytes  received 19 bytes  170.00 bytes/sec
2026-03-09 10:53:01.717080 | controller-0 | total size is 0  speedup is 0.00
2026-03-09 14:53:01.863051 | hotstack -> 38.102.83.163 | changed
2026-03-09 14:53:01.871383 | 
2026-03-09 14:53:01.871490 | LOOP [hotlogs : Collect hotstack data]
2026-03-09 14:53:02.527577 | hotstack -> 38.102.83.210 | changed: >f+++++++++ .openshift_install.log
2026-03-09 14:53:02.527822 | 
2026-03-09 14:53:02.967782 | hotstack -> 38.102.83.210 | changed:
2026-03-09 14:53:02.967898 | hotstack -> 38.102.83.210 | .d..t...... ./
2026-03-09 14:53:02.967925 | hotstack -> 38.102.83.210 | cd+++++++++ butane/
2026-03-09 14:53:02.967946 | hotstack -> 38.102.83.210 | >f+++++++++ butane/90-master-disable-netifnames.bu
2026-03-09 14:53:02.967965 | hotstack -> 38.102.83.210 | >f+++++++++ butane/90-master-enable-iscsi.bu
2026-03-09 14:53:02.967984 | hotstack -> 38.102.83.210 | >f+++++++++ butane/91-master-enable-multipath.bu
2026-03-09 14:53:02.968003 | hotstack -> 38.102.83.210 | >f+++++++++ butane/92-master-lv-cinder-volumes.bu
2026-03-09 14:53:02.968021 | hotstack -> 38.102.83.210 | cd+++++++++ config-assets/
2026-03-09 14:53:02.968038 | hotstack -> 38.102.83.210 | >f+++++++++ config-assets/95-etcd_config.yaml
2026-03-09 14:53:02.968056 | hotstack -> 38.102.83.210 | >f+++++++++ config-assets/ovn_k8s_config.yaml
2026-03-09 14:53:02.968072 | hotstack -> 38.102.83.210 | cd+++++++++ machine-configs/
2026-03-09 14:53:02.968090 | hotstack -> 38.102.83.210 | >f+++++++++ machine-configs/90-master-disable-netifnames.yaml
2026-03-09 14:53:02.968107 | hotstack -> 38.102.83.210 | >f+++++++++ machine-configs/90-master-enable-iscsi.yaml
2026-03-09 14:53:02.968125 | hotstack -> 38.102.83.210 | >f+++++++++ machine-configs/91-master-enable-multipath.yaml
2026-03-09 14:53:02.968142 | hotstack -> 38.102.83.210 | >f+++++++++ machine-configs/92-master-lv-cinder-volumes.yaml
2026-03-09 14:53:03.452957 | hotstack -> 38.102.83.210 | changed:
2026-03-09 14:53:03.453070 | hotstack -> 38.102.83.210 | .d..t...... ./
2026-03-09 14:53:03.453095 | hotstack -> 38.102.83.210 | >f+++++++++ ironic_nodes.yaml
2026-03-09 14:53:03.903122 | hotstack -> 38.102.83.210 | changed:
2026-03-09 14:53:03.903241 | hotstack -> 38.102.83.210 | .d..t...... ./
2026-03-09 14:53:03.903266 | hotstack -> 38.102.83.210 | cd+++++++++ sushy_emulator_manifests/
2026-03-09 14:53:03.903290 | hotstack -> 38.102.83.210 | >f+++++++++ sushy_emulator_manifests/config_map.yaml.applied
2026-03-09 14:53:03.903318 | hotstack -> 38.102.83.210 | >f+++++++++ sushy_emulator_manifests/config_map.yaml.applied.log
2026-03-09 14:53:03.903336 | hotstack -> 38.102.83.210 | >f+++++++++ sushy_emulator_manifests/deployment.yaml.applied
2026-03-09 14:53:03.903356 | hotstack -> 38.102.83.210 | >f+++++++++ sushy_emulator_manifests/deployment.yaml.applied.log
2026-03-09 14:53:03.903374 | hotstack -> 38.102.83.210 | >f+++++++++ sushy_emulator_manifests/namespace.yaml.applied
2026-03-09 14:53:03.903391 | hotstack -> 38.102.83.210 | >f+++++++++ sushy_emulator_manifests/namespace.yaml.applied.log
2026-03-09 14:53:03.903407 | hotstack -> 38.102.83.210 | >f+++++++++ sushy_emulator_manifests/service.yaml.applied
2026-03-09 14:53:03.903424 | hotstack -> 38.102.83.210 | >f+++++++++ sushy_emulator_manifests/service.yaml.applied.log
2026-03-09 14:53:04.465387 | hotstack -> 38.102.83.210 | ERROR: Item: {'src': '/home/zuul/must-gather.tar.gz', 'dest_dir': '.'}
2026-03-09 14:53:04.465952 | hotstack -> 38.102.83.210 | {
2026-03-09 14:53:04.465997 | hotstack -> 38.102.83.210 |   "ansible_loop_var": "item",
2026-03-09 14:53:04.466034 | hotstack -> 38.102.83.210 |   "item": {
2026-03-09 14:53:04.466054 | hotstack -> 38.102.83.210 |     "dest_dir": ".",
2026-03-09 14:53:04.466071 | hotstack -> 38.102.83.210 |     "src": "/home/zuul/must-gather.tar.gz"
2026-03-09 14:53:04.466088 | hotstack -> 38.102.83.210 |   },
2026-03-09 14:53:04.466107 | hotstack -> 38.102.83.210 |   "msg": "Warning: Permanently added '38.102.83.163' (ED25519) to the list of known hosts.\r\nrsync: [sender] link_stat \"/home/zuul/must-gather.tar.gz\" failed: No such file or directory (2)\nrsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1841) [Receiver=3.2.5]\nrsync: [Receiver] write error: Broken pipe (32)\n",
2026-03-09 14:53:04.466128 | hotstack -> 38.102.83.210 |   "rc": 23
2026-03-09 14:53:04.466145 | hotstack -> 38.102.83.210 | }
2026-03-09 14:53:04.925021 | hotstack -> 38.102.83.210 | changed: .d..t...... ./
2026-03-09 14:53:05.379393 | hotstack -> 38.102.83.210 | changed: .d..t...... ./
2026-03-09 14:53:05.402901 | hotstack | ERROR: Ignoring Errors
2026-03-09 14:53:05.423859 | 
2026-03-09 14:53:05.423996 | LOOP [hotlogs : Check hotstack data collection failures]
2026-03-09 14:53:05.500861 | hotstack | skipping: Conditional result was False
2026-03-09 14:53:05.508613 | hotstack | skipping: Conditional result was False
2026-03-09 14:53:05.515441 | hotstack | skipping: Conditional result was False
2026-03-09 14:53:05.522457 | hotstack | skipping: Conditional result was False
2026-03-09 14:53:05.545206 | hotstack | ERROR
2026-03-09 14:53:05.545455 | hotstack | {
2026-03-09 14:53:05.545525 | hotstack |   "msg": "Unexpected templating type error occurred on ({{ hotlogs_failures + ['Failed to collect: ' + item.item] }}): can only concatenate str (not \"dict\") to str. can only concatenate str (not \"dict\") to str"
2026-03-09 14:53:05.545572 | hotstack | }
2026-03-09 14:53:05.560799 | 
2026-03-09 14:53:05.561089 | TASK [Print message that log collection failed]
2026-03-09 14:53:05.600611 | hotstack | ok: WARNING: Log collection failed and may be incomplete. Continue with cleanup ...
2026-03-09 14:53:05.635273 | 
2026-03-09 14:53:05.635408 | PLAY RECAP
2026-03-09 14:53:05.635503 | hotstack | ok: 15 changed: 7 unreachable: 0 failed: 0 skipped: 1 rescued: 2 ignored: 1
2026-03-09 14:53:05.635540 | 
2026-03-09 14:53:05.762669 | POST-RUN END RESULT_NORMAL: [trusted : review.rdoproject.org/config/playbooks/hotstack/post.yaml@master]
2026-03-09 14:53:05.765604 | POST-RUN START: [trusted : review.rdoproject.org/config/playbooks/hotstack/post-clean.yaml@master]
2026-03-09 14:53:06.365064 | 
2026-03-09 14:53:06.365186 | PLAY [Clean up hotstack environment]
2026-03-09 14:53:06.388701 | 
2026-03-09 14:53:06.388851 | TASK [Set _auth_type fact]
2026-03-09 14:53:06.428497 | hotstack | Output suppressed because no_log was given
2026-03-09 14:53:06.434614 | 
2026-03-09 14:53:06.434702 | TASK [Write clouds.yaml]
2026-03-09 14:53:07.544839 | hotstack | Output suppressed because no_log was given
2026-03-09 14:53:07.561555 | 
2026-03-09 14:53:07.561671 | TASK [Delete application credentials]
2026-03-09 14:53:09.607054 | hotstack | changed
2026-03-09 14:53:09.621001 | 
2026-03-09 14:53:09.621166 | TASK [Print message that application credential failed to delete]
2026-03-09 14:53:09.648770 | hotstack | skipping: Conditional result was False
2026-03-09 14:53:09.660325 | 
2026-03-09 14:53:09.660468 | TASK [Print message that application credential deleted successfully]
2026-03-09 14:53:09.698068 | hotstack | ok: Application credential deleted successfully.
2026-03-09 14:53:09.709188 | 
2026-03-09 14:53:09.709330 | TASK [Write clouds.yaml with application credentials]
2026-03-09 14:53:09.734465 | 
2026-03-09 14:53:09.734673 | TASK [Delete the heat stack]
2026-03-09 10:53:42.005543 | hotstack | 2026-03-09 14:53:11Z [hs-5ad27496488549c98262fbfcd2662052]: DELETE_IN_PROGRESS  Stack DELETE started
2026-03-09 10:53:42.005629 | hotstack | 2026-03-09 14:53:12Z [hs-5ad27496488549c98262fbfcd2662052.ironic0]: DELETE_IN_PROGRESS  state changed
2026-03-09 10:53:42.005638 | hotstack | 2026-03-09 14:53:13Z [hs-5ad27496488549c98262fbfcd2662052.master0]: DELETE_IN_PROGRESS  state changed
2026-03-09 10:53:42.005645 | hotstack | 2026-03-09 14:53:14Z [hs-5ad27496488549c98262fbfcd2662052.ironic-net-router-interface]: DELETE_IN_PROGRESS  state changed
2026-03-09 10:53:42.005652 | hotstack | 2026-03-09 14:53:15Z [hs-5ad27496488549c98262fbfcd2662052.controller]: DELETE_IN_PROGRESS  state changed
2026-03-09 10:53:42.005658 | hotstack | 2026-03-09 14:53:16Z [hs-5ad27496488549c98262fbfcd2662052.controller-floating-ip]: DELETE_IN_PROGRESS  state changed
2026-03-09 10:53:42.005665 | hotstack | 2026-03-09 14:53:16Z [hs-5ad27496488549c98262fbfcd2662052.ironic-net-router-interface]: DELETE_COMPLETE  state changed
2026-03-09 10:53:42.005669 | hotstack | 2026-03-09 14:53:17Z [hs-5ad27496488549c98262fbfcd2662052.ctlplane-net-router-interface]: DELETE_IN_PROGRESS  state changed
2026-03-09 10:53:42.005673 | hotstack | 2026-03-09 14:53:18Z [hs-5ad27496488549c98262fbfcd2662052.controller-floating-ip]: DELETE_COMPLETE  state changed
2026-03-09 10:53:42.005679 | hotstack | 2026-03-09 14:53:18Z [hs-5ad27496488549c98262fbfcd2662052.machine-net-router-interface]: DELETE_IN_PROGRESS  state changed
2026-03-09 10:53:42.005684 | hotstack | 2026-03-09 14:53:19Z [hs-5ad27496488549c98262fbfcd2662052.ctlplane-net-router-interface]: DELETE_COMPLETE  state changed
2026-03-09 10:53:42.005691 | hotstack | 2026-03-09 14:53:20Z [hs-5ad27496488549c98262fbfcd2662052.machine-net-router-interface]: DELETE_COMPLETE  state changed
2026-03-09 10:53:42.005697 | hotstack | 2026-03-09 14:53:20Z [hs-5ad27496488549c98262fbfcd2662052.router]: DELETE_IN_PROGRESS  state changed
2026-03-09 10:53:42.005703 | hotstack | 2026-03-09 14:53:22Z [hs-5ad27496488549c98262fbfcd2662052.router]: DELETE_COMPLETE  state changed
2026-03-09 10:53:42.005708 | hotstack | 2026-03-09 14:53:22Z [hs-5ad27496488549c98262fbfcd2662052.controller]: DELETE_COMPLETE  state changed
2026-03-09 10:53:42.005712 | hotstack | 2026-03-09 14:53:22Z [hs-5ad27496488549c98262fbfcd2662052.controller-init]: DELETE_IN_PROGRESS  state changed
2026-03-09 10:53:42.005717 | hotstack | 2026-03-09 14:53:22Z [hs-5ad27496488549c98262fbfcd2662052.controller-init]: DELETE_COMPLETE  state changed
2026-03-09 10:53:42.005722 | hotstack | 2026-03-09 14:53:22Z [hs-5ad27496488549c98262fbfcd2662052.controller_users]: DELETE_IN_PROGRESS  state changed
2026-03-09 10:53:42.005726 | hotstack | 2026-03-09 14:53:22Z [hs-5ad27496488549c98262fbfcd2662052.controller-runcmd]: DELETE_IN_PROGRESS  state changed
2026-03-09 10:53:42.005730 | hotstack | 2026-03-09 14:53:22Z [hs-5ad27496488549c98262fbfcd2662052.controller-write-files]: DELETE_IN_PROGRESS  state changed
2026-03-09 10:53:42.005735 | hotstack | 2026-03-09 14:53:22Z [hs-5ad27496488549c98262fbfcd2662052.controller_users]: DELETE_COMPLETE  state changed
2026-03-09 10:53:42.005739 | hotstack | 2026-03-09 14:53:22Z [hs-5ad27496488549c98262fbfcd2662052.controller-runcmd]: DELETE_COMPLETE  state changed
2026-03-09 10:53:42.005743 | hotstack | 2026-03-09 14:53:22Z [hs-5ad27496488549c98262fbfcd2662052.controller-write-files]: DELETE_COMPLETE  state changed
2026-03-09 10:53:42.005748 | hotstack | 2026-03-09 14:53:23Z [hs-5ad27496488549c98262fbfcd2662052.ironic0]: DELETE_COMPLETE  state changed
2026-03-09 10:53:42.005753 | hotstack | 2026-03-09 14:53:23Z [hs-5ad27496488549c98262fbfcd2662052.ironic0-port]: DELETE_IN_PROGRESS  state changed
2026-03-09 10:53:42.005757 | hotstack | 2026-03-09 14:53:23Z [hs-5ad27496488549c98262fbfcd2662052.ironic0-port]: DELETE_COMPLETE  state changed
2026-03-09 10:53:42.005761 | hotstack | 2026-03-09 14:53:35Z [hs-5ad27496488549c98262fbfcd2662052.master0]: DELETE_COMPLETE  state changed
2026-03-09 10:53:42.005766 | hotstack | 2026-03-09 14:53:35Z [hs-5ad27496488549c98262fbfcd2662052.master0-lvms-vol0]: DELETE_IN_PROGRESS  state changed
2026-03-09 10:53:42.005770 | hotstack | 2026-03-09 14:53:35Z [hs-5ad27496488549c98262fbfcd2662052.master0-cinder-vol2]: DELETE_IN_PROGRESS  state changed
2026-03-09 10:53:42.005774 | hotstack | 2026-03-09 14:53:35Z [hs-5ad27496488549c98262fbfcd2662052.master0-machine-port]: DELETE_IN_PROGRESS  state changed
2026-03-09 10:53:42.005785 | hotstack | 2026-03-09 14:53:35Z [hs-5ad27496488549c98262fbfcd2662052.master0-trunk0]: DELETE_IN_PROGRESS  state changed
2026-03-09 10:53:42.005789 | hotstack | 2026-03-09 14:53:35Z [hs-5ad27496488549c98262fbfcd2662052.master0-ironic-port]: DELETE_IN_PROGRESS  state changed
2026-03-09 10:53:42.005796 | hotstack | 2026-03-09 14:53:35Z [hs-5ad27496488549c98262fbfcd2662052.master0-cinder-vol1]: DELETE_IN_PROGRESS  state changed
2026-03-09 10:53:42.005801 | hotstack | 2026-03-09 14:53:35Z [hs-5ad27496488549c98262fbfcd2662052.master0-cinder-vol0]: DELETE_IN_PROGRESS  state changed
2026-03-09 10:53:42.005805 | hotstack | 2026-03-09 14:53:35Z [hs-5ad27496488549c98262fbfcd2662052.master0-machine-port]: DELETE_COMPLETE  state changed
2026-03-09 10:53:42.005809 | hotstack | 2026-03-09 14:53:35Z [hs-5ad27496488549c98262fbfcd2662052.extra-dhcp-opts-value]: DELETE_IN_PROGRESS  state changed
2026-03-09 10:53:42.005819 | hotstack | 2026-03-09 14:53:35Z [hs-5ad27496488549c98262fbfcd2662052.extra-dhcp-opts-value]: DELETE_COMPLETE  state changed
2026-03-09 10:53:42.005878 | hotstack | 2026-03-09 14:53:36Z [hs-5ad27496488549c98262fbfcd2662052.controller-machine-port]: DELETE_IN_PROGRESS  state changed
2026-03-09 10:53:42.005885 | hotstack | 2026-03-09 14:53:36Z [hs-5ad27496488549c98262fbfcd2662052.master0-cinder-vol2]: DELETE_COMPLETE  state changed
2026-03-09 10:53:42.005890 | hotstack | 2026-03-09 14:53:36Z [hs-5ad27496488549c98262fbfcd2662052.master0-trunk0]: DELETE_COMPLETE  state changed
2026-03-09 10:53:42.005895 | hotstack | 2026-03-09 14:53:36Z [hs-5ad27496488549c98262fbfcd2662052.master0-tenant-port]: DELETE_IN_PROGRESS  state changed
2026-03-09 10:53:42.005901 | hotstack | 2026-03-09 14:53:36Z [hs-5ad27496488549c98262fbfcd2662052.master0-storage-port]: DELETE_IN_PROGRESS  state changed
2026-03-09 10:53:42.005905 | hotstack | 2026-03-09 14:53:36Z [hs-5ad27496488549c98262fbfcd2662052.master0-lvms-vol0]: DELETE_COMPLETE  state changed
2026-03-09 10:53:42.005910 | hotstack | 2026-03-09 14:53:36Z [hs-5ad27496488549c98262fbfcd2662052.master0-ctlplane-trunk-parent-port]: DELETE_IN_PROGRESS  state changed
2026-03-09 10:53:42.005915 | hotstack | 2026-03-09 14:53:36Z [hs-5ad27496488549c98262fbfcd2662052.master0-cinder-vol1]: DELETE_COMPLETE  state changed
2026-03-09 10:53:42.005920 | hotstack | 2026-03-09 14:53:36Z [hs-5ad27496488549c98262fbfcd2662052.master0-internal-api-port]: DELETE_IN_PROGRESS  state changed
2026-03-09 10:53:42.005924 | hotstack | 2026-03-09 14:53:36Z [hs-5ad27496488549c98262fbfcd2662052.master0-ironic-port]: DELETE_COMPLETE  state changed
2026-03-09 10:53:42.005932 | hotstack | 2026-03-09 14:53:36Z [hs-5ad27496488549c98262fbfcd2662052.ironic-subnet]: DELETE_IN_PROGRESS  state changed
2026-03-09 10:53:42.005937 | hotstack | 2026-03-09 14:53:36Z [hs-5ad27496488549c98262fbfcd2662052.master0-cinder-vol0]: DELETE_COMPLETE  state changed
2026-03-09 10:53:42.005942 | hotstack | 2026-03-09 14:53:36Z [hs-5ad27496488549c98262fbfcd2662052.controller-machine-port]: DELETE_COMPLETE  state changed
2026-03-09 10:53:42.005947 | hotstack | 2026-03-09 14:53:36Z [hs-5ad27496488549c98262fbfcd2662052.machine-subnet]: DELETE_IN_PROGRESS  state changed
2026-03-09 10:53:42.005951 | hotstack | 2026-03-09 14:53:37Z [hs-5ad27496488549c98262fbfcd2662052.master0-tenant-port]: DELETE_COMPLETE  state changed
2026-03-09 10:53:42.005957 | hotstack | 2026-03-09 14:53:37Z [hs-5ad27496488549c98262fbfcd2662052.tenant-subnet]: DELETE_IN_PROGRESS  state changed
2026-03-09 10:53:42.005964 | hotstack | 2026-03-09 14:53:37Z [hs-5ad27496488549c98262fbfcd2662052.master0-ctlplane-trunk-parent-port]: DELETE_COMPLETE  state changed
2026-03-09 10:53:42.072181 | hotstack | 2026-03-09 14:53:37Z [hs-5ad27496488549c98262fbfcd2662052.ctlplane-subnet]: DELETE_IN_PROGRESS  state changed
2026-03-09 10:53:42.072244 | hotstack | 2026-03-09 14:53:37Z [hs-5ad27496488549c98262fbfcd2662052.ironic-subnet]: DELETE_COMPLETE  state changed
2026-03-09 10:53:42.072250 | hotstack | 2026-03-09 14:53:37Z [hs-5ad27496488549c98262fbfcd2662052.ironic-net]: DELETE_IN_PROGRESS  state changed
2026-03-09 10:53:42.072255 | hotstack | 2026-03-09 14:53:37Z [hs-5ad27496488549c98262fbfcd2662052.tenant-subnet]: DELETE_COMPLETE  state changed
2026-03-09 10:53:42.072342 | hotstack | 2026-03-09 14:53:37Z [hs-5ad27496488549c98262fbfcd2662052.tenant-net]: DELETE_IN_PROGRESS  state changed
2026-03-09 10:53:42.072362 | hotstack | 2026-03-09 14:53:37Z [hs-5ad27496488549c98262fbfcd2662052.machine-subnet]: DELETE_COMPLETE  state changed
2026-03-09 10:53:42.072367 | hotstack | 2026-03-09 14:53:37Z [hs-5ad27496488549c98262fbfcd2662052.machine-net]: DELETE_IN_PROGRESS  state changed
2026-03-09 10:53:42.072372 | hotstack | 2026-03-09 14:53:37Z [hs-5ad27496488549c98262fbfcd2662052.ctlplane-subnet]: DELETE_COMPLETE  state changed
2026-03-09 10:53:42.072376 | hotstack | 2026-03-09 14:53:37Z [hs-5ad27496488549c98262fbfcd2662052.ctlplane-net]: DELETE_IN_PROGRESS  state changed
2026-03-09 10:53:42.072381 | hotstack | 2026-03-09 14:53:38Z [hs-5ad27496488549c98262fbfcd2662052.master0-storage-port]: DELETE_COMPLETE  state changed
2026-03-09 10:53:42.072396 | hotstack | 2026-03-09 14:53:38Z [hs-5ad27496488549c98262fbfcd2662052.storage-subnet]: DELETE_IN_PROGRESS  state changed
2026-03-09 10:53:42.072401 | hotstack | 2026-03-09 14:53:38Z [hs-5ad27496488549c98262fbfcd2662052.master0-internal-api-port]: DELETE_COMPLETE  state changed
2026-03-09 10:53:42.072406 | hotstack | 2026-03-09 14:53:38Z [hs-5ad27496488549c98262fbfcd2662052.internal-api-subnet]: DELETE_IN_PROGRESS  state changed
2026-03-09 10:53:42.072411 | hotstack | 2026-03-09 14:53:38Z [hs-5ad27496488549c98262fbfcd2662052.ironic-net]: DELETE_COMPLETE  state changed
2026-03-09 10:53:42.072415 | hotstack | 2026-03-09 14:53:39Z [hs-5ad27496488549c98262fbfcd2662052.machine-net]: DELETE_COMPLETE  state changed
2026-03-09 10:53:42.072420 | hotstack | 2026-03-09 14:53:39Z [hs-5ad27496488549c98262fbfcd2662052.tenant-net]: DELETE_COMPLETE  state changed
2026-03-09 10:53:42.072424 | hotstack | 2026-03-09 14:53:39Z [hs-5ad27496488549c98262fbfcd2662052.internal-api-subnet]: DELETE_COMPLETE  state changed
2026-03-09 10:53:42.072429 | hotstack | 2026-03-09 14:53:39Z [hs-5ad27496488549c98262fbfcd2662052.internal-api-net]: DELETE_IN_PROGRESS  state changed
2026-03-09 10:53:42.072433 | hotstack | 2026-03-09 14:53:39Z [hs-5ad27496488549c98262fbfcd2662052.storage-subnet]: DELETE_COMPLETE  state changed
2026-03-09 10:53:42.072438 | hotstack | 2026-03-09 14:53:39Z [hs-5ad27496488549c98262fbfcd2662052.storage-net]: DELETE_IN_PROGRESS  state changed
2026-03-09 10:53:42.072442 | hotstack | 2026-03-09 14:53:39Z [hs-5ad27496488549c98262fbfcd2662052.ctlplane-net]: DELETE_COMPLETE  state changed
2026-03-09 10:53:42.072446 | hotstack | 2026-03-09 14:53:40Z [hs-5ad27496488549c98262fbfcd2662052.internal-api-net]: DELETE_COMPLETE  state changed
2026-03-09 10:53:42.072451 | hotstack | 2026-03-09 14:53:40Z [hs-5ad27496488549c98262fbfcd2662052.storage-net]: DELETE_COMPLETE  state changed
2026-03-09 10:53:42.072456 | hotstack | 2026-03-09 14:53:40Z [hs-5ad27496488549c98262fbfcd2662052]: DELETE_COMPLETE  Stack DELETE completed successfully
2026-03-09 14:53:42.319245 | hotstack | changed
2026-03-09 14:53:42.331455 | 
2026-03-09 14:53:42.331568 | TASK [Print message that stack failed to delete]
2026-03-09 14:53:42.356814 | hotstack | skipping: Conditional result was False
2026-03-09 14:53:42.370328 | 
2026-03-09 14:53:42.370719 | TASK [Print message that stack deleted successfully]
2026-03-09 14:53:42.422303 | hotstack | ok: Stack deleted successfully.
2026-03-09 14:53:42.432842 | 
2026-03-09 14:53:42.432974 | TASK [Delete clouds.yaml]
2026-03-09 14:53:42.817418 | hotstack | Output suppressed because no_log was given
2026-03-09 14:53:42.849424 | 
2026-03-09 14:53:42.849542 | PLAY RECAP
2026-03-09 14:53:42.849598 | hotstack | ok: 7 changed: 4 unreachable: 0 failed: 0 skipped: 3 rescued: 0 ignored: 0
2026-03-09 14:53:42.849627 | 
2026-03-09 14:53:42.965358 | POST-RUN END RESULT_NORMAL: [trusted : review.rdoproject.org/config/playbooks/hotstack/post-clean.yaml@master]
2026-03-09 14:53:42.975151 | POST-RUN START: [trusted : review.rdoproject.org/config/playbooks/base-minimal/post-ssh.yaml@master]
2026-03-09 14:53:43.542159 | 
2026-03-09 14:53:43.542276 | PLAY [all]
2026-03-09 14:53:43.561618 | 
2026-03-09 14:53:43.561789 | TASK [include_role : fetch-output]
2026-03-09 14:53:43.623169 | hotstack | ok
2026-03-09 14:53:43.641672 | 
2026-03-09 14:53:43.641813 | TASK [fetch-output : Set log path for multiple nodes]
2026-03-09 14:53:43.696861 | hotstack | skipping: Conditional result was False
2026-03-09 14:53:43.703452 | 
2026-03-09 14:53:43.703532 | TASK [fetch-output : Set log path for single node]
2026-03-09 14:53:43.770255 | hotstack | ok
2026-03-09 14:53:43.780232 | 
2026-03-09 14:53:43.780328 | LOOP [fetch-output : Ensure local output dirs]
2026-03-09 14:53:44.235758 | hotstack -> localhost | ok: "/var/lib/zuul/builds/5ad27496488549c98262fbfcd2662052/work/logs"
2026-03-09 14:53:44.498787 | hotstack -> localhost | changed: "/var/lib/zuul/builds/5ad27496488549c98262fbfcd2662052/work/artifacts"
2026-03-09 14:53:44.751409 | hotstack -> localhost | changed: "/var/lib/zuul/builds/5ad27496488549c98262fbfcd2662052/work/docs"
2026-03-09 14:53:44.774006 | 
2026-03-09 14:53:44.774236 | LOOP [fetch-output : Collect logs, artifacts and docs]
2026-03-09 14:53:45.463771 | hotstack | changed:
2026-03-09 14:53:45.464621 | hotstack | .d..t...... ./
2026-03-09 14:53:45.464799 | hotstack | cd+++++++++ hotlogs/
2026-03-09 14:53:45.464871 | hotstack | cd+++++++++ hotlogs/data/
2026-03-09 14:53:45.464916 | hotstack | >f+++++++++ hotlogs/data/ironic_nodes.yaml
2026-03-09 14:53:45.464990 | hotstack | cd+++++++++ hotlogs/manifests/
2026-03-09 14:53:45.465032 | hotstack | cd+++++++++ hotlogs/manifests/sushy_emulator_manifests/
2026-03-09 14:53:45.465072 | hotstack | >f+++++++++ hotlogs/manifests/sushy_emulator_manifests/config_map.yaml.applied
2026-03-09 14:53:45.465109 | hotstack | >f+++++++++ hotlogs/manifests/sushy_emulator_manifests/config_map.yaml.applied.log
2026-03-09 14:53:45.465147 | hotstack | >f+++++++++ hotlogs/manifests/sushy_emulator_manifests/deployment.yaml.applied
2026-03-09 14:53:45.465183 | hotstack | >f+++++++++ hotlogs/manifests/sushy_emulator_manifests/deployment.yaml.applied.log
2026-03-09 14:53:45.465219 | hotstack | >f+++++++++ hotlogs/manifests/sushy_emulator_manifests/namespace.yaml.applied
2026-03-09 14:53:45.465254 | hotstack | >f+++++++++ hotlogs/manifests/sushy_emulator_manifests/namespace.yaml.applied.log
2026-03-09 14:53:45.465290 | hotstack | >f+++++++++ hotlogs/manifests/sushy_emulator_manifests/service.yaml.applied
2026-03-09 14:53:45.465325 | hotstack | >f+++++++++ hotlogs/manifests/sushy_emulator_manifests/service.yaml.applied.log
2026-03-09 14:53:45.465360 | hotstack | cd+++++++++ hotlogs/nova-console-recordings/
2026-03-09 14:53:45.465418 | hotstack | cd+++++++++ hotlogs/ocp_cluster/
2026-03-09 14:53:45.465458 | hotstack | >f+++++++++ hotlogs/ocp_cluster/openshift_install.log
2026-03-09 14:53:45.465495 | hotstack | cd+++++++++ hotlogs/ocp_cluster/butane/
2026-03-09 14:53:45.465532 | hotstack | >f+++++++++ hotlogs/ocp_cluster/butane/90-master-disable-netifnames.bu
2026-03-09 14:53:45.465567 | hotstack | >f+++++++++ hotlogs/ocp_cluster/butane/90-master-enable-iscsi.bu
2026-03-09 14:53:45.465603 | hotstack | >f+++++++++ hotlogs/ocp_cluster/butane/91-master-enable-multipath.bu
2026-03-09 14:53:45.465639 | hotstack | >f+++++++++ hotlogs/ocp_cluster/butane/92-master-lv-cinder-volumes.bu
2026-03-09 14:53:45.465674 | hotstack | cd+++++++++ hotlogs/ocp_cluster/config-assets/
2026-03-09 14:53:45.465708 | hotstack | >f+++++++++ hotlogs/ocp_cluster/config-assets/95-etcd_config.yaml
2026-03-09 14:53:45.465808 | hotstack | >f+++++++++ hotlogs/ocp_cluster/config-assets/ovn_k8s_config.yaml
2026-03-09 14:53:45.465868 | hotstack | cd+++++++++ hotlogs/ocp_cluster/machine-configs/
2026-03-09 14:53:45.465911 | hotstack | >f+++++++++ hotlogs/ocp_cluster/machine-configs/90-master-disable-netifnames.yaml
2026-03-09 14:53:45.465947 | hotstack | >f+++++++++ hotlogs/ocp_cluster/machine-configs/90-master-enable-iscsi.yaml
2026-03-09 14:53:45.465981 | hotstack | >f+++++++++ hotlogs/ocp_cluster/machine-configs/91-master-enable-multipath.yaml
2026-03-09 14:53:45.466016 | hotstack | >f+++++++++ hotlogs/ocp_cluster/machine-configs/92-master-lv-cinder-volumes.yaml
2026-03-09 14:53:45.466101 | hotstack | changed: All items complete
2026-03-09 14:53:45.466164 | 
2026-03-09 14:53:45.993229 | hotstack | changed: .d..t...... ./
2026-03-09 14:53:46.497253 | hotstack | changed: .d..t...... ./
2026-03-09 14:53:46.519132 | 
2026-03-09 14:53:46.519289 | TASK [include_role : fetch-output-openshift]
2026-03-09 14:53:46.545789 | hotstack | skipping: Conditional result was False
2026-03-09 14:53:46.553984 | 
2026-03-09 14:53:46.554083 | LOOP [merge-output-to-logs : Move artifacts and docs to logs dir]
2026-03-09 14:53:47.003079 | hotstack -> localhost | ok: Item: artifacts Runtime: 0:00:00.013707
2026-03-09 14:53:47.293495 | hotstack -> localhost | ok: Item: docs Runtime: 0:00:00.007375
2026-03-09 14:53:47.344144 | 
2026-03-09 14:53:47.344334 | PLAY [all]
2026-03-09 14:53:47.366759 | 
2026-03-09 14:53:47.366928 | TASK [remove-build-sshkey : Remove the build SSH key from all nodes]
2026-03-09 14:53:47.979348 | hotstack | changed
2026-03-09 14:53:48.028700 | 
2026-03-09 14:53:48.028846 | PLAY RECAP
2026-03-09 14:53:48.028909 | hotstack | ok: 5 changed: 4 unreachable: 0 failed: 0 skipped: 2 rescued: 0 ignored: 0
2026-03-09 14:53:48.028941 | 
2026-03-09 14:53:48.145909 | POST-RUN END RESULT_NORMAL: [trusted : review.rdoproject.org/config/playbooks/base-minimal/post-ssh.yaml@master]
2026-03-09 14:53:48.155191 | POST-RUN START: [trusted : review.rdoproject.org/config/playbooks/base-minimal/post-logs.yaml@master]
2026-03-09 14:53:48.842633 | 
2026-03-09 14:53:48.842858 | PLAY [localhost]
2026-03-09 14:53:48.861839 | 
2026-03-09 14:53:48.861969 | TASK [generate-zuul-manifest : Generate Zuul manifest]
2026-03-09 14:53:49.331140 | localhost | changed
2026-03-09 14:53:49.336508 | 
2026-03-09 14:53:49.336589 | TASK [generate-zuul-manifest : Return Zuul manifest URL to Zuul]
2026-03-09 14:53:49.389868 | localhost | ok
2026-03-09 14:53:49.398288 | 
2026-03-09 14:53:49.398363 | TASK [add-fileserver : Create SSH private key tempfile]
2026-03-09 14:53:49.821270 | localhost | changed
2026-03-09 14:53:49.834142 | 
2026-03-09 14:53:49.834293 | TASK [add-fileserver : Create SSH private key from secret]
2026-03-09 14:53:50.759458 | localhost | changed
2026-03-09 14:53:50.772432 | 
2026-03-09 14:53:50.772592 | TASK [add-fileserver : Add fileserver ssh key]
2026-03-09 14:53:51.288768 | localhost | Identity added: /var/lib/zuul/builds/5ad27496488549c98262fbfcd2662052/work/tmp/ansible.278db95e (/var/lib/zuul/builds/5ad27496488549c98262fbfcd2662052/work/tmp/ansible.278db95e)
2026-03-09 14:53:51.289035 | localhost | ok: Runtime: 0:00:00.012184
2026-03-09 14:53:51.296462 | 
2026-03-09 14:53:51.296551 | TASK [add-fileserver : Remove SSH private key from disk]
2026-03-09 14:53:51.606418 | localhost | ok: Runtime: 0:00:00.006117
2026-03-09 14:53:51.618443 | 
2026-03-09 14:53:51.618590 | TASK [add-fileserver : Add fileserver to inventory]
2026-03-09 14:53:51.705932 | localhost | changed
2026-03-09 14:53:51.712779 | 
2026-03-09 14:53:51.712882 | TASK [add-fileserver : Add fileserver server to known hosts]
2026-03-09 14:53:52.163043 | localhost | changed
2026-03-09 14:53:52.190386 | 
2026-03-09 14:53:52.190512 | PLAY [localhost]
2026-03-09 14:53:52.237551 | 
2026-03-09 14:53:52.237683 | TASK [Generate bulk log download script]
2026-03-09 14:53:52.270006 | localhost | ok
2026-03-09 14:53:52.285785 | 
2026-03-09 14:53:52.285912 | TASK [local-log-download : Check API endpoint is defined]
2026-03-09 14:53:52.325322 | localhost | ok: All assertions passed
2026-03-09 14:53:52.336297 | 
2026-03-09 14:53:52.336424 | TASK [local-log-download : Create download script]
2026-03-09 14:53:52.800471 | localhost -> localhost | changed
2026-03-09 14:53:52.820234 | 
2026-03-09 14:53:52.820417 | TASK [Register quick-download link]
2026-03-09 14:53:52.858642 | localhost | ok
2026-03-09 14:53:52.891344 | 
2026-03-09 14:53:52.891532 | PLAY [logserver.rdoproject.org]
2026-03-09 14:53:52.908876 | 
2026-03-09 14:53:52.909046 | TASK [Set zuul-log-path fact]
2026-03-09 14:53:52.941251 | logserver.rdoproject.org | ok
2026-03-09 14:53:52.964010 | 
2026-03-09 14:53:52.964206 | TASK [set-zuul-log-path-fact : Set log path for a build]
2026-03-09 14:53:53.009210 | logserver.rdoproject.org | ok
2026-03-09 14:53:53.022581 | 
2026-03-09 14:53:53.022805 | TASK [upload-logs : Create log directories]
2026-03-09 14:53:54.039325 | logserver.rdoproject.org | changed
2026-03-09 14:53:54.044615 | 
2026-03-09 14:53:54.044763 | TASK [upload-logs : Ensure logs are readable before uploading]
2026-03-09 14:53:54.402373 | logserver.rdoproject.org -> localhost | ok: Runtime: 0:00:00.007440
2026-03-09 14:53:54.408097 | 
2026-03-09 14:53:54.408172 | TASK [upload-logs : Upload logs to log server]
2026-03-09 14:53:55.170388 | logserver.rdoproject.org | Output suppressed because no_log was given
2026-03-09 14:53:55.173816 | 
2026-03-09 14:53:55.173882 | LOOP [upload-logs : Compress console log and json output]
2026-03-09 14:53:55.243921 | logserver.rdoproject.org | skipping: Conditional result was False
2026-03-09 14:53:55.259106 | logserver.rdoproject.org | skipping: Conditional result was False
2026-03-09 14:53:55.264032 | 
2026-03-09 14:53:55.264195 | LOOP [upload-logs : Upload compressed console log and json output]
2026-03-09 14:53:55.315232 | logserver.rdoproject.org | skipping: Conditional result was False
2026-03-09 14:53:55.315565 | 
2026-03-09 14:53:55.319002 | logserver.rdoproject.org | skipping: Conditional result was False
2026-03-09 14:53:55.323817 | 
2026-03-09 14:53:55.323975 | LOOP [upload-logs : Upload console log and json output]