2025-10-11 12:18:42.063004 | Job console starting...
2025-10-11 12:18:42.078385 | Updating repositories
2025-10-11 12:18:42.109813 | Preparing job workspace
2025-10-11 12:18:47.391303 | Running Ansible setup...
2025-10-11 12:18:51.421814 | PRE-RUN START: [trusted : review.rdoproject.org/config/playbooks/base-minimal/pre.yaml@master]
2025-10-11 12:18:51.988149 | 
2025-10-11 12:18:51.988293 | PLAY [localhost]
2025-10-11 12:18:51.996582 | 
2025-10-11 12:18:51.996647 | TASK [Gathering Facts]
2025-10-11 12:18:52.925092 | localhost | ok
2025-10-11 12:18:52.942996 | 
2025-10-11 12:18:52.943102 | TASK [log-inventory : Ensure Zuul Ansible directory exists]
2025-10-11 12:18:53.295553 | localhost -> localhost | changed
2025-10-11 12:18:53.306789 | 
2025-10-11 12:18:53.306929 | TASK [log-inventory : Copy ansible inventory to logs dir]
2025-10-11 12:18:54.161257 | localhost -> localhost | changed
2025-10-11 12:18:54.178558 | 
2025-10-11 12:18:54.178687 | TASK [Setup log path fact]
2025-10-11 12:18:54.204116 | localhost | ok
2025-10-11 12:18:54.227455 | 
2025-10-11 12:18:54.227589 | TASK [set-zuul-log-path-fact : Set log path for a build]
2025-10-11 12:18:54.271355 | localhost | ok
2025-10-11 12:18:54.287142 | 
2025-10-11 12:18:54.287275 | TASK [emit-job-header : Print job information]
2025-10-11 12:18:54.341183 | # Job Information
2025-10-11 12:18:54.341395 | Ansible Version: 2.15.12
2025-10-11 12:18:54.341450 | Job: vexxhost-hotstack-secret-rotate
2025-10-11 12:18:54.341492 | Pipeline: github-experimental-trigger
2025-10-11 12:18:54.341531 | Executor: ze03.softwarefactory-project.io
2025-10-11 12:18:54.341568 | Triggered by: https://github.com/openstack-k8s-operators/hotstack/pull/256
2025-10-11 12:18:54.341607 | Log URL (when completed): https://logserver.rdoproject.org/2ee/rdoproject.org/2eeebd5c21844fdd9eefbaf9d59e8d1b/
2025-10-11 12:18:54.341646 | Event ID: 540c5a10-a69c-11f0-919c-892b7c3c53d4
2025-10-11 12:18:54.349266 | 
2025-10-11 12:18:54.349399 | LOOP [emit-job-header : Print node information]
2025-10-11 12:18:54.475497 | localhost | ok:
2025-10-11 12:18:54.475829 | localhost | # Node Information
2025-10-11 12:18:54.475889 | localhost | Inventory Hostname: hotstack
2025-10-11 12:18:54.475938 | localhost | Hostname: np0005481195
2025-10-11 12:18:54.475979 | localhost | Username: zuul
2025-10-11 12:18:54.476021 | localhost | Distro: CentOS 9
2025-10-11 12:18:54.476058 | localhost | Provider: vexxhost-nodepool-tripleo
2025-10-11 12:18:54.476095 | localhost | Region: RegionOne
2025-10-11 12:18:54.476130 | localhost | Label: cloud-centos-9-stream-tripleo-vexxhost
2025-10-11 12:18:54.476167 | localhost | Product Name: OpenStack Nova
2025-10-11 12:18:54.476203 | localhost | Interface IP: 38.102.83.97
2025-10-11 12:18:54.518395 | 
2025-10-11 12:18:54.518532 | PLAY [all]
2025-10-11 12:18:54.531956 | 
2025-10-11 12:18:54.532083 | TASK [Gather network facts]
2025-10-11 12:18:55.091396 | hotstack | ok
2025-10-11 12:18:55.123105 | 
2025-10-11 12:18:55.123259 | TASK [include_role : start-zuul-console]
2025-10-11 12:18:55.148394 | hotstack | ok
2025-10-11 12:18:55.171607 | 
2025-10-11 12:18:55.171689 | TASK [start-zuul-console : Start zuul_console daemon.]
2025-10-11 12:18:55.634010 | hotstack | ok
2025-10-11 12:18:55.653941 | 
2025-10-11 12:18:55.654075 | TASK [include_role : add-build-sshkey]
2025-10-11 12:18:55.680165 | hotstack | ok
2025-10-11 12:18:55.708973 | 
2025-10-11 12:18:55.709059 | TASK [add-build-sshkey : Check to see if ssh key was already created for this build]
2025-10-11 12:18:55.967026 | hotstack -> localhost | ok
2025-10-11 12:18:55.978226 | 
2025-10-11 12:18:55.978350 | TASK [add-build-sshkey : Create a new key in workspace based on build UUID]
2025-10-11 12:18:56.002824 | hotstack | ok
2025-10-11 12:18:56.030150 | hotstack | included: /var/lib/zuul/builds/2eeebd5c21844fdd9eefbaf9d59e8d1b/trusted/project_1/opendev.org/zuul/zuul-jobs/roles/add-build-sshkey/tasks/create-key-and-replace.yaml
2025-10-11 12:18:56.038087 | 
2025-10-11 12:18:56.038170 | TASK [add-build-sshkey : Create Temp SSH key]
2025-10-11 12:18:56.888236 | hotstack -> localhost | Generating public/private rsa key pair.
2025-10-11 12:18:56.888521 | hotstack -> localhost | Your identification has been saved in /var/lib/zuul/builds/2eeebd5c21844fdd9eefbaf9d59e8d1b/work/2eeebd5c21844fdd9eefbaf9d59e8d1b_id_rsa.
2025-10-11 12:18:56.888589 | hotstack -> localhost | Your public key has been saved in /var/lib/zuul/builds/2eeebd5c21844fdd9eefbaf9d59e8d1b/work/2eeebd5c21844fdd9eefbaf9d59e8d1b_id_rsa.pub.
2025-10-11 12:18:56.888636 | hotstack -> localhost | The key fingerprint is:
2025-10-11 12:18:56.888678 | hotstack -> localhost | SHA256:1bDdjPZnDhxAiw92i5kMEi+N4c3jI+3lik9qvxk0cJc zuul-build-sshkey
2025-10-11 12:18:56.888719 | hotstack -> localhost | The key's randomart image is:
2025-10-11 12:18:56.888817 | hotstack -> localhost | +---[RSA 3072]----+
2025-10-11 12:18:56.888864 | hotstack -> localhost | |      o   ..o    |
2025-10-11 12:18:56.888904 | hotstack -> localhost | |     . O   B *   |
2025-10-11 12:18:56.888943 | hotstack -> localhost | |      * O E B +  |
2025-10-11 12:18:56.888981 | hotstack -> localhost | |       B B O + . |
2025-10-11 12:18:56.889019 | hotstack -> localhost | |      . S * o + o|
2025-10-11 12:18:56.889055 | hotstack -> localhost | |       + =     = |
2025-10-11 12:18:56.889091 | hotstack -> localhost | |        + .     .|
2025-10-11 12:18:56.889128 | hotstack -> localhost | |      .+ +       |
2025-10-11 12:18:56.889164 | hotstack -> localhost | |     .oo*.       |
2025-10-11 12:18:56.889200 | hotstack -> localhost | +----[SHA256]-----+
2025-10-11 12:18:56.889276 | hotstack -> localhost | ok: Runtime: 0:00:00.412968
2025-10-11 12:18:56.901959 | 
2025-10-11 12:18:56.902091 | TASK [add-build-sshkey : Remote setup ssh keys (linux)]
2025-10-11 12:18:56.938225 | hotstack | ok
2025-10-11 12:18:56.957933 | hotstack | included: /var/lib/zuul/builds/2eeebd5c21844fdd9eefbaf9d59e8d1b/trusted/project_1/opendev.org/zuul/zuul-jobs/roles/add-build-sshkey/tasks/remote-linux.yaml
2025-10-11 12:18:56.972512 | 
2025-10-11 12:18:56.972594 | TASK [add-build-sshkey : Remove previously added zuul-build-sshkey]
2025-10-11 12:18:56.997126 | hotstack | skipping: Conditional result was False
2025-10-11 12:18:57.005123 | 
2025-10-11 12:18:57.005215 | TASK [add-build-sshkey : Enable access via build key on all nodes]
2025-10-11 12:18:57.486295 | hotstack | changed
2025-10-11 12:18:57.493395 | 
2025-10-11 12:18:57.493478 | TASK [add-build-sshkey : Make sure user has a .ssh]
2025-10-11 12:18:57.783359 | hotstack | ok
2025-10-11 12:18:57.794281 | 
2025-10-11 12:18:57.794412 | TASK [add-build-sshkey : Install build private key as SSH key on all nodes]
2025-10-11 12:18:58.831070 | hotstack | changed
2025-10-11 12:18:58.842018 | 
2025-10-11 12:18:58.842177 | TASK [add-build-sshkey : Install build public key as SSH key on all nodes]
2025-10-11 12:18:59.829087 | hotstack | changed
2025-10-11 12:18:59.839534 | 
2025-10-11 12:18:59.839664 | TASK [add-build-sshkey : Remote setup ssh keys (windows)]
2025-10-11 12:18:59.866384 | hotstack | skipping: Conditional result was False
2025-10-11 12:18:59.877964 | 
2025-10-11 12:18:59.878101 | TASK [remove-zuul-sshkey : Remove master key from local agent]
2025-10-11 12:19:00.301865 | hotstack -> localhost | changed
2025-10-11 12:19:00.323082 | 
2025-10-11 12:19:00.323221 | TASK [add-build-sshkey : Add back temp key]
2025-10-11 12:19:00.655323 | hotstack -> localhost | Identity added: /var/lib/zuul/builds/2eeebd5c21844fdd9eefbaf9d59e8d1b/work/2eeebd5c21844fdd9eefbaf9d59e8d1b_id_rsa (zuul-build-sshkey)
2025-10-11 12:19:00.655817 | hotstack -> localhost | ok: Runtime: 0:00:00.013981
2025-10-11 12:19:00.667850 | 
2025-10-11 12:19:00.667986 | TASK [add-build-sshkey : Verify we can still SSH to all nodes]
2025-10-11 12:19:01.094231 | hotstack | ok
2025-10-11 12:19:01.139228 | 
2025-10-11 12:19:01.139380 | TASK [add-build-sshkey : Verify we can still SSH to all nodes (windows)]
2025-10-11 12:19:01.175051 | hotstack | skipping: Conditional result was False
2025-10-11 12:19:01.196343 | 
2025-10-11 12:19:01.196489 | TASK [include_role : validate-host]
2025-10-11 12:19:01.233805 | hotstack | ok
2025-10-11 12:19:01.269741 | 
2025-10-11 12:19:01.269858 | TASK [validate-host : Define zuul_info_dir fact]
2025-10-11 12:19:01.314115 | hotstack | ok
2025-10-11 12:19:01.324998 | 
2025-10-11 12:19:01.325116 | TASK [validate-host : Ensure Zuul Ansible directory exists]
2025-10-11 12:19:01.615600 | hotstack -> localhost | ok
2025-10-11 12:19:01.621305 | 
2025-10-11 12:19:01.621372 | TASK [validate-host : Collect information about the host]
2025-10-11 12:19:02.469628 | hotstack | ok
2025-10-11 12:19:02.487766 | 
2025-10-11 12:19:02.487898 | TASK [validate-host : Sanitize hostname]
2025-10-11 12:19:02.565100 | hotstack | ok
2025-10-11 12:19:02.576621 | 
2025-10-11 12:19:02.576794 | TASK [validate-host : Write out all ansible variables/facts known for each host]
2025-10-11 12:19:03.114661 | hotstack -> localhost | changed
2025-10-11 12:19:03.121339 | 
2025-10-11 12:19:03.121418 | TASK [validate-host : Collect information about zuul worker]
2025-10-11 12:19:03.599062 | hotstack | ok
2025-10-11 12:19:03.609863 | 
2025-10-11 12:19:03.609999 | TASK [validate-host : Write out all zuul information for each host]
2025-10-11 12:19:04.115184 | hotstack -> localhost | changed
2025-10-11 12:19:04.124895 | 
2025-10-11 12:19:04.124965 | TASK [include_role : prepare-workspace-openshift]
2025-10-11 12:19:04.138370 | hotstack | skipping: Conditional result was False
2025-10-11 12:19:04.144552 | 
2025-10-11 12:19:04.144624 | TASK [include_role : remove-zuul-sshkey]
2025-10-11 12:19:04.157913 | hotstack | skipping: Conditional result was False
2025-10-11 12:19:04.165562 | 
2025-10-11 12:19:04.165660 | LOOP [ensure-output-dirs : Empty Zuul Output directories by removing them]
2025-10-11 12:19:04.473416 | hotstack | ok: "logs"
2025-10-11 12:19:04.473772 | hotstack | ok: All items complete
2025-10-11 12:19:04.473830 | 
2025-10-11 12:19:04.736879 | hotstack | ok: "artifacts"
2025-10-11 12:19:05.000673 | hotstack | ok: "docs"
2025-10-11 12:19:05.020454 | 
2025-10-11 12:19:05.020637 | LOOP [ensure-output-dirs : Ensure Zuul Output directories exist]
2025-10-11 12:19:05.336163 | hotstack | changed: "logs"
2025-10-11 12:19:05.584060 | hotstack | changed: "artifacts"
2025-10-11 12:19:05.835149 | hotstack | changed: "docs"
2025-10-11 12:19:05.897703 | 
2025-10-11 12:19:05.897854 | PLAY RECAP
2025-10-11 12:19:05.897911 | hotstack | ok: 22 changed: 9 unreachable: 0 failed: 0 skipped: 5 rescued: 0 ignored: 0
2025-10-11 12:19:05.897948 | localhost | ok: 6 changed: 2 unreachable: 0 failed: 0 skipped: 0 rescued: 0 ignored: 0
2025-10-11 12:19:05.897974 | 
2025-10-11 12:19:06.011920 | PRE-RUN END RESULT_NORMAL: [trusted : review.rdoproject.org/config/playbooks/base-minimal/pre.yaml@master]
2025-10-11 12:19:06.017498 | PRE-RUN START: [trusted : review.rdoproject.org/config/playbooks/hotstack/pre_stage1.yaml@master]
2025-10-11 12:19:06.624216 | 
2025-10-11 12:19:06.624326 | PLAY [Create necessary files to deploy hotstack scenario]
2025-10-11 12:19:06.645110 | 
2025-10-11 12:19:06.645196 | TASK [prepare-workspace : Start zuul_console daemon.]
2025-10-11 12:19:07.074804 | hotstack | ok
2025-10-11 12:19:07.087478 | 
2025-10-11 12:19:07.087644 | TASK [prepare-workspace : Synchronize src repos to workspace directory.]
2025-10-11 12:19:09.796376 | hotstack | Output suppressed because no_log was given
2025-10-11 12:19:09.814188 | 
2025-10-11 12:19:09.814303 | LOOP [add-authorized-keys : Enable access via build key on all nodes]
2025-10-11 12:19:10.298923 | hotstack | changed:
2025-10-11 12:19:10.299323 | hotstack | {
2025-10-11 12:19:10.299392 | hotstack |   "public_key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCINPOs6XsfMVQMqjmiduT3Bxhogy7MbAFOWlR+9LlKkTW1kV6glsCXY5svtggd2zRtc40UWalHaj9jC+yEkZdI35h2PkK+Sww52u4nczqgNpIFNt7Dm2xtYKbTolAV9jSeNkRlGMCq9EBzQlwiV0HkHx1CdNR/zzXbyNZLmOybFkXqR150H01kUhdv0v6AMWTDZjMEWO/PaWCOgzZhH9ztXjy+rwa7pdaI4+0Pb9Kib+d8/IrgNsLB/sCazqD+9/COI6AHqzTDS1aU9/gMQrUN1ronzMEQhzzOuoI8WeLqU++j+0dT2UWVPlQqoln+RYu7dUn/DSZDrsYenudT5+P"
2025-10-11 12:19:10.299442 | hotstack | }
2025-10-11 12:19:10.331343 | 
2025-10-11 12:19:10.331523 | TASK [Assert cloud name and cloud secret]
2025-10-11 12:19:10.403584 | hotstack | Output suppressed because no_log was given
2025-10-11 12:19:10.419394 | 
2025-10-11 12:19:10.419524 | TASK [Ensure openstack release repo is installed]
2025-10-11 12:19:15.724906 | hotstack | changed
2025-10-11 12:19:15.735871 | 
2025-10-11 12:19:15.736002 | TASK [Install openstack client packages]
2025-10-11 12:19:31.748085 | hotstack | changed
2025-10-11 12:19:31.759690 | 
2025-10-11 12:19:31.759849 | TASK [Ensure HotStack work dir]
2025-10-11 12:19:32.227326 | hotstack | changed
2025-10-11 12:19:32.238658 | 
2025-10-11 12:19:32.238831 | LOOP [Ensure etc directories]
2025-10-11 12:19:32.590174 | hotstack | changed: "/etc/openstack"
2025-10-11 12:19:32.590585 | hotstack | changed: All items complete
2025-10-11 12:19:32.590643 | 
2025-10-11 12:19:32.887453 | hotstack | changed: "/etc/hotstack"
2025-10-11 12:19:32.904284 | 
2025-10-11 12:19:32.904515 | TASK [Ensure .ssh dir exists]
2025-10-11 12:19:33.198898 | hotstack | Output suppressed because no_log was given
2025-10-11 12:19:33.210452 | 
2025-10-11 12:19:33.210624 | TASK [Ensure ssh keypair exists]
2025-10-11 12:19:43.780500 | hotstack | Output suppressed because no_log was given
2025-10-11 12:19:43.786293 | 
2025-10-11 12:19:43.786376 | TASK [Generate CA cert file]
2025-10-11 12:19:43.820697 | hotstack | skipping: Conditional result was False
2025-10-11 12:19:43.826923 | 
2025-10-11 12:19:43.827035 | TASK [Update CA certs]
2025-10-11 12:19:44.356796 | hotstack | skipping: Conditional result was False
2025-10-11 12:19:44.368848 | 
2025-10-11 12:19:44.368988 | TASK [Set _auth_type fact]
2025-10-11 12:19:44.406451 | hotstack | Output suppressed because no_log was given
2025-10-11 12:19:44.416495 | 
2025-10-11 12:19:44.416626 | TASK [Write clouds.yaml - user/password]
2025-10-11 12:19:45.652569 | hotstack | Output suppressed because no_log was given
2025-10-11 12:19:45.663426 | 
2025-10-11 12:19:45.663586 | TASK [Create application credential for sushy-emulator]
2025-10-11 12:19:48.214222 | hotstack | Output suppressed because no_log was given
2025-10-11 12:19:48.225080 | 
2025-10-11 12:19:48.225242 | TASK [Read application credential from file]
2025-10-11 12:19:48.591416 | hotstack | Output suppressed because no_log was given
2025-10-11 12:19:48.602354 | 
2025-10-11 12:19:48.602524 | TASK [Delete temporary application credential file]
2025-10-11 12:19:48.877197 | hotstack | Output suppressed because no_log was given
2025-10-11 12:19:48.887807 | 
2025-10-11 12:19:48.887988 | TASK [Assert application credential id and secret]
2025-10-11 12:19:48.976868 | hotstack | Output suppressed because no_log was given
2025-10-11 12:19:48.987302 | 
2025-10-11 12:19:48.987470 | TASK [Write new clouds.yaml with application credentials]
2025-10-11 12:19:50.017411 | hotstack | Output suppressed because no_log was given
2025-10-11 12:19:50.027236 | 
2025-10-11 12:19:50.027360 | TASK [Write hotstack cloud-secret.yaml]
2025-10-11 12:19:50.941927 | hotstack | Output suppressed because no_log was given
2025-10-11 12:19:50.952847 | 
2025-10-11 12:19:50.952977 | TASK [Write clouds.yaml with application credentials]
2025-10-11 12:19:50.981989 | 
2025-10-11 12:19:50.982320 | TASK [Write hotstack cloud-secret.yaml]
2025-10-11 12:19:51.076019 | 
2025-10-11 12:19:51.076149 | PLAY RECAP
2025-10-11 12:19:51.076188 | hotstack | ok: 18 changed: 11 unreachable: 0 failed: 0 skipped: 4 rescued: 0 ignored: 0
2025-10-11 12:19:51.076208 | 
2025-10-11 12:19:51.167851 | PRE-RUN END RESULT_NORMAL: [trusted : review.rdoproject.org/config/playbooks/hotstack/pre_stage1.yaml@master]
2025-10-11 12:19:51.173380 | PRE-RUN START: [trusted : review.rdoproject.org/config/playbooks/hotstack/pre_stage2.yaml@master]
2025-10-11 12:19:51.773670 | 
2025-10-11 12:19:51.773793 | PLAY [Create hotstack scenario]
2025-10-11 12:19:51.794290 | 
2025-10-11 12:19:51.794366 | TASK [Assert scenario is defined]
2025-10-11 12:19:51.876554 | hotstack | ok: All assertions passed
2025-10-11 12:19:51.884049 | 
2025-10-11 12:19:51.884147 | TASK [Fetch bootstrap vars from remote to zuul executor]
2025-10-11 12:19:52.419528 | hotstack | Output suppressed because no_log was given
2025-10-11 12:19:52.430478 | 
2025-10-11 12:19:52.430619 | TASK [Load bootstrap vars with template resolution]
2025-10-11 12:19:52.467557 | hotstack | Output suppressed because no_log was given
2025-10-11 12:19:52.478537 | 
2025-10-11 12:19:52.478673 | TASK [Fetch automation vars from remote to zuul executor]
2025-10-11 12:19:52.857333 | hotstack | Output suppressed because no_log was given
2025-10-11 12:19:52.863617 | 
2025-10-11 12:19:52.863699 | TASK [Load automation vars with template resolution]
2025-10-11 12:19:52.945053 | hotstack | Output suppressed because no_log was given
2025-10-11 12:19:52.949793 | 
2025-10-11 12:19:52.949857 | TASK [Slurp zuul-build-sshkey from file]
2025-10-11 12:19:53.360904 | hotstack | Output suppressed because no_log was given
2025-10-11 12:19:53.372935 | 
2025-10-11 12:19:53.373097 | TASK [Set up infrastructure (Heat stack)]
2025-10-11 12:19:53.404053 | hotstack | Output suppressed because no_log was given
2025-10-11 12:19:53.459619 | 
2025-10-11 12:19:53.459720 | TASK [dataplane_ssh_keys : Ensure Dataplane ssh keypair exists]
2025-10-11 12:19:54.088783 | hotstack | changed
2025-10-11 12:19:54.100582 | 
2025-10-11 12:19:54.100714 | TASK [dataplane_ssh_keys : Ensure Nova migration ssh keypair exists]
2025-10-11 12:19:54.538948 | hotstack | changed
2025-10-11 12:19:54.551873 | 
2025-10-11 12:19:54.552013 | TASK [dataplane_ssh_keys : Set SSH public key facts]
2025-10-11 12:19:54.591709 | hotstack | ok
2025-10-11 12:19:54.604850 | 
2025-10-11 12:19:54.604990 | TASK [dataplane_ssh_keys : Store SSH public key facts in file]
2025-10-11 12:19:55.694760 | hotstack | changed
2025-10-11 12:19:55.704884 | 
2025-10-11 12:19:55.704957 | TASK [heat_stack : Assert config is defined]
2025-10-11 12:19:55.805138 | hotstack | ok: All assertions passed
2025-10-11 12:19:55.810057 | 
2025-10-11 12:19:55.810119 | TASK [heat_stack : Get latest snapset and update stack parameters]
2025-10-11 12:19:55.855432 | hotstack | skipping: Conditional result was False
2025-10-11 12:19:55.862321 | 
2025-10-11 12:19:55.862403 | TASK [heat_stack : Debug latest snapset information]
2025-10-11 12:19:55.927063 | hotstack | skipping: Conditional result was False
2025-10-11 12:19:55.933065 | 
2025-10-11 12:19:55.933159 | TASK [heat_stack : Create stack]
2025-10-11 12:21:21.618422 | hotstack | changed
2025-10-11 12:21:21.629973 | 
2025-10-11 12:21:21.630056 | TASK [heat_stack : Wait for stack to complete]
2025-10-11 12:21:23.634915 | hotstack | ok
2025-10-11 12:21:23.648251 | 
2025-10-11 12:21:23.648453 | TASK [heat_stack : Debug - stack_info]
2025-10-11 12:21:23.675631 | hotstack | skipping: Conditional result was False
2025-10-11 12:21:23.688252 | 
2025-10-11 12:21:23.688414 | TASK [heat_stack : Stop if stack not complete]
2025-10-11 12:21:23.715323 | hotstack | skipping: Conditional result was False
2025-10-11 12:21:23.729136 | 
2025-10-11 12:21:23.729322 | TASK [heat_stack : Get stack outputs]
2025-10-11 12:21:25.744890 | hotstack | changed
2025-10-11 12:21:25.761535 | 
2025-10-11 12:21:25.761670 | TASK [heat_stack : Set stack outputs facts]
2025-10-11 12:21:25.811808 | hotstack | ok
2025-10-11 12:21:25.827318 | 
2025-10-11 12:21:25.827453 | TASK [heat_stack : Store stack outputs in file]
2025-10-11 12:21:26.785172 | hotstack | changed
2025-10-11 12:21:26.798116 | 
2025-10-11 12:21:26.798204 | TASK [Fetch stack outputs from remote to zuul executor]
2025-10-11 12:21:27.190633 | hotstack | Output suppressed because no_log was given
2025-10-11 12:21:27.201319 | 
2025-10-11 12:21:27.201454 | TASK [Load stack output vars with template resolution]
2025-10-11 12:21:27.242932 | hotstack | Output suppressed because no_log was given
2025-10-11 12:21:27.253358 | 
2025-10-11 12:21:27.253510 | TASK [Fetch cloud secrets from remote to zuul executor]
2025-10-11 12:21:27.647651 | hotstack | Output suppressed because no_log was given
2025-10-11 12:21:27.658223 | 
2025-10-11 12:21:27.658384 | TASK [Load hotstack cloud secret vars with template resolution]
2025-10-11 12:21:27.707025 | hotstack | Output suppressed because no_log was given
2025-10-11 12:21:27.717463 | 
2025-10-11 12:21:27.717581 | TASK [Add Hotstack controller-0 to ansible inventory]
2025-10-11 12:21:27.767254 | hotstack | Output suppressed because no_log was given
2025-10-11 12:21:27.778458 | 
2025-10-11 12:21:27.778581 | TASK [Wait for Hotstack controller-0 to be ready]
2025-10-11 12:21:38.274479 | hotstack -> 38.102.83.20 | Output suppressed because no_log was given
2025-10-11 12:21:38.295033 | 
2025-10-11 12:21:38.295316 | TASK [Start Zuul console on Hotstack controller-0]
2025-10-11 12:21:38.329227 | hotstack | Output suppressed because no_log was given
2025-10-11 12:21:38.372374 | 
2025-10-11 12:21:38.372817 | TASK [start-zuul-console : Start zuul_console daemon.]
2025-10-11 12:21:38.891277 | hotstack -> 38.102.83.20 | ok
2025-10-11 12:21:38.917856 | 
2025-10-11 12:21:38.918112 | TASK [Prepare the controller node]
2025-10-11 12:21:39.011834 | hotstack | Output suppressed because no_log was given
2025-10-11 12:21:39.048916 | 
2025-10-11 12:21:39.049078 | TASK [controller : Assert config is defined]
2025-10-11 12:21:39.131485 | hotstack | ok: All assertions passed
2025-10-11 12:21:39.138658 | 
2025-10-11 12:21:39.138808 | TASK [controller : Add controller-0 to the Ansible inventory]
2025-10-11 12:21:39.170206 | hotstack | ok
2025-10-11 12:21:39.175572 | 
2025-10-11 12:21:39.175634 | TASK [controller : Read Dataplane SSH private key content]
2025-10-11 12:21:39.466545 | hotstack | ok
2025-10-11 12:21:39.472507 | 
2025-10-11 12:21:39.472593 | TASK [controller : Read Nova migration SSH private key content]
2025-10-11 12:21:39.755920 | hotstack | ok
2025-10-11 12:21:39.770936 | 
2025-10-11 12:21:39.771170 | TASK [controller : Wait for controller-0 to be ready]
2025-10-11 12:21:40.373583 | hotstack -> 38.102.83.20 | ok
2025-10-11 12:21:40.383404 | 
2025-10-11 12:21:40.383551 | LOOP [controller : Ensure directories exists]
2025-10-11 12:21:40.905880 | hotstack -> 38.102.83.20 | changed: "/home/zuul/data"
2025-10-11 12:21:41.198511 | hotstack -> 38.102.83.20 | changed: "/home/zuul/bin"
2025-10-11 12:21:41.217102 | 
2025-10-11 12:21:41.217369 | LOOP [controller : Add common functions to the bin_dir]
2025-10-11 12:21:42.048449 | hotstack -> 38.102.83.20 | changed: "bin/hotstack-common-funcs.sh"
2025-10-11 12:21:42.049022 | hotstack -> 38.102.83.20 | changed: All items complete
2025-10-11 12:21:42.049105 | 
2025-10-11 12:21:42.069448 | 
2025-10-11 12:21:42.069782 | LOOP [controller : Add bin utilities to the bin_dir]
2025-10-11 12:21:42.965977 | hotstack -> 38.102.83.20 | changed: "bin/hotstack-approve-install-plan"
2025-10-11 12:21:44.680057 | hotstack -> 38.102.83.20 | changed: "bin/hotstack-leader-election-tune"
2025-10-11 12:21:45.913814 | hotstack -> 38.102.83.20 | changed: "bin/hotstack-nova-discover-hosts"
2025-10-11 12:21:47.109057 | hotstack -> 38.102.83.20 | changed: "bin/hotstack-snapset"
2025-10-11 12:21:48.307410 | hotstack -> 38.102.83.20 | changed: "bin/hotstack-openstack-version-patch"
2025-10-11 12:21:49.497768 | hotstack -> 38.102.83.20 | changed: "bin/hotstack-wait-for-bmh"
2025-10-11 12:21:49.520659 | 
2025-10-11 12:21:49.520913 | TASK [controller : Write ansible inventory to file on controller-0]
2025-10-11 12:21:49.599187 | hotstack | skipping: Conditional result was False
2025-10-11 12:21:49.611940 | 
2025-10-11 12:21:49.612098 | TASK [controller : Copy Dataplane ssh private key to controller]
2025-10-11 12:21:50.915788 | hotstack -> 38.102.83.20 | changed
2025-10-11 12:21:50.930889 | 
2025-10-11 12:21:50.931045 | TASK [controller : Copy Dataplane ssh public key to controller]
2025-10-11 12:21:52.214087 | hotstack -> 38.102.83.20 | changed
2025-10-11 12:21:52.228107 | 
2025-10-11 12:21:52.228245 | TASK [controller : Copy Nova migrate ssh private key to controller]
2025-10-11 12:21:53.542140 | hotstack -> 38.102.83.20 | changed
2025-10-11 12:21:53.556260 | 
2025-10-11 12:21:53.556397 | TASK [controller : Copy Nova migrate ssh public key to controller]
2025-10-11 12:21:54.815173 | hotstack -> 38.102.83.20 | changed
2025-10-11 12:21:54.828366 | 
2025-10-11 12:21:54.828540 | TASK [controller : Copy dataplane authorized keys to controller]
2025-10-11 12:21:56.067544 | hotstack -> 38.102.83.20 | changed
2025-10-11 12:21:56.082493 | 
2025-10-11 12:21:56.082636 | TASK [controller : Ensure cloud config directory exists]
2025-10-11 12:21:56.488286 | hotstack -> 38.102.83.20 | changed
2025-10-11 12:21:56.502045 | 
2025-10-11 12:21:56.502186 | TASK [controller : Template the clouds.yaml]
2025-10-11 12:21:57.293064 | hotstack -> 38.102.83.20 | Output suppressed because no_log was given
2025-10-11 12:21:57.306947 | 
2025-10-11 12:21:57.307093 | TASK [controller : Write Ironic nodes YAML]
2025-10-11 12:21:57.355049 | hotstack | skipping: Conditional result was False
2025-10-11 12:21:57.379172 | 
2025-10-11 12:21:57.379389 | TASK [Install Openshift]
2025-10-11 12:21:57.413250 | hotstack | Output suppressed because no_log was given
2025-10-11 12:21:57.470407 | 
2025-10-11 12:21:57.470491 | TASK [ocp_agent_installer : Assert config is defined]
2025-10-11 12:21:57.555627 | hotstack -> 38.102.83.20 | ok: All assertions passed
2025-10-11 12:21:57.568894 | 
2025-10-11 12:21:57.569034 | TASK [ocp_agent_installer : Install using agent installer]
2025-10-11 12:21:57.629877 | hotstack | ok
2025-10-11 12:21:57.667494 | hotstack | included: /var/lib/zuul/builds/2eeebd5c21844fdd9eefbaf9d59e8d1b/trusted/project_3/github.com/openstack-k8s-operators/hotstack/roles/ocp_agent_installer/tasks/_install.yml
2025-10-11 12:21:57.676132 | 
2025-10-11 12:21:57.676213 | TASK [ocp_agent_installer : Assert config is defined]
2025-10-11 12:21:57.762237 | hotstack -> 38.102.83.20 | ok: All assertions passed
2025-10-11 12:21:57.775539 | 
2025-10-11 12:21:57.775685 | LOOP [ocp_agent_installer : Ensure directory exists]
2025-10-11 12:21:58.165993 | hotstack -> 38.102.83.20 | ok: "/home/zuul/bin"
2025-10-11 12:21:58.166441 | hotstack -> 38.102.83.20 | changed: All items complete
2025-10-11 12:21:58.166501 | 
2025-10-11 12:21:58.493430 | hotstack -> 38.102.83.20 | changed: "/home/zuul/.kube"
2025-10-11 12:21:58.794925 | hotstack -> 38.102.83.20 | changed: "/home/zuul/ocp-cluster"
2025-10-11 12:21:59.165496 | hotstack -> 38.102.83.20 | changed: "/home/zuul/ocp-cluster/openshift"
2025-10-11 12:21:59.437759 | hotstack -> 38.102.83.20 | changed: "/home/zuul/agent-installer"
2025-10-11 12:21:59.728213 | hotstack -> 38.102.83.20 | changed: "/home/zuul/cluster-custom-config/"
2025-10-11 12:22:00.038077 | hotstack -> 38.102.83.20 | changed: "/home/zuul/cluster-custom-config//butane"
2025-10-11 12:22:00.344526 | hotstack -> 38.102.83.20 | changed: "/home/zuul/cluster-custom-config//machine-configs"
2025-10-11 12:22:00.632293 | hotstack -> 38.102.83.20 | changed: "/home/zuul/cluster-custom-config//config-assets"
2025-10-11 12:22:00.647104 | 
2025-10-11 12:22:00.647218 | TASK [ocp_agent_installer : Install package requirements for agent installer]
2025-10-11 12:22:20.096674 | hotstack -> 38.102.83.20 | ok: Nothing to do
2025-10-11 12:22:20.105847 | 
2025-10-11 12:22:20.105938 | TASK [ocp_agent_installer : Run tasks/install_client.yml]
2025-10-11 12:22:20.142919 | hotstack | ok
2025-10-11 12:22:20.158895 | hotstack | included: /var/lib/zuul/builds/2eeebd5c21844fdd9eefbaf9d59e8d1b/trusted/project_3/github.com/openstack-k8s-operators/hotstack/roles/ocp_agent_installer/tasks/install_client.yml
2025-10-11 12:22:20.176400 | 
2025-10-11 12:22:20.176486 | TASK [ocp_agent_installer : Download the client]
2025-10-11 12:22:22.097976 | hotstack -> 38.102.83.20 | changed: OK (73347743 bytes)
2025-10-11 12:22:22.112811 | 
2025-10-11 12:22:22.112980 | TASK [ocp_agent_installer : Extract client to /home/zuul/bin]
2025-10-11 12:22:26.775155 | hotstack -> 38.102.83.20 | changed
2025-10-11 12:22:26.789496 | 
2025-10-11 12:22:26.789638 | TASK [ocp_agent_installer : Configure bash completion]
2025-10-11 12:22:27.825001 | hotstack -> 38.102.83.20 | changed
2025-10-11 12:22:27.837896 | 
2025-10-11 12:22:27.838037 | TASK [ocp_agent_installer : Run tasks/install_installer.yml]
2025-10-11 12:22:27.875530 | hotstack | ok
2025-10-11 12:22:27.891644 | hotstack | included: /var/lib/zuul/builds/2eeebd5c21844fdd9eefbaf9d59e8d1b/trusted/project_3/github.com/openstack-k8s-operators/hotstack/roles/ocp_agent_installer/tasks/install_installer.yml
2025-10-11 12:22:27.914868 | 
2025-10-11 12:22:27.914983 | TASK [ocp_agent_installer : Download the installer]
2025-10-11 12:22:36.795343 | hotstack -> 38.102.83.20 | changed: OK (437684516 bytes)
2025-10-11 12:22:36.808011 | 
2025-10-11 12:22:36.808139 | TASK [ocp_agent_installer : Extract installer to /home/zuul/bin]
2025-10-11 12:22:51.235090 | hotstack -> 38.102.83.20 | changed
2025-10-11 12:22:51.248698 | 
2025-10-11 12:22:51.248900 | TASK [ocp_agent_installer : Write OCP install-config.yaml]
2025-10-11 12:22:52.153528 | hotstack -> 38.102.83.20 | changed
2025-10-11 12:22:52.160300 | 
2025-10-11 12:22:52.160395 | TASK [ocp_agent_installer : Replace pull secret in install-config.yaml]
2025-10-11 12:22:52.695672 | hotstack -> 38.102.83.20 | Output suppressed because no_log was given
2025-10-11 12:22:52.710565 | 
2025-10-11 12:22:52.710718 | TASK [ocp_agent_installer : Write OCP agent-config.yaml]
2025-10-11 12:22:53.570409 | hotstack -> 38.102.83.20 | changed
2025-10-11 12:22:53.583204 | 
2025-10-11 12:22:53.583338 | TASK [ocp_agent_installer : Include tasks to generate machine configs]
2025-10-11 12:22:53.622014 | hotstack | ok
2025-10-11 12:22:53.641457 | hotstack | included: /var/lib/zuul/builds/2eeebd5c21844fdd9eefbaf9d59e8d1b/trusted/project_3/github.com/openstack-k8s-operators/hotstack/roles/ocp_agent_installer/tasks/machine_configs.yml
2025-10-11 12:22:53.663227 | 
2025-10-11 12:22:53.663316 | LOOP [ocp_agent_installer : Template butane config for net.ifnames]
2025-10-11 12:22:54.528406 | hotstack -> 38.102.83.20 | changed: "master"
2025-10-11 12:22:54.550973 | 
2025-10-11 12:22:54.551186 | LOOP [ocp_agent_installer : Generate MachineConfig for iscsi]
2025-10-11 12:22:55.097089 | hotstack -> 38.102.83.20 | changed
2025-10-11 12:22:55.097544 | hotstack -> 38.102.83.20 | changed: All items complete
2025-10-11 12:22:55.097601 | 
2025-10-11 12:22:55.111618 | 
2025-10-11 12:22:55.111865 | LOOP [ocp_agent_installer : Template butane config for iscsi]
2025-10-11 12:22:55.962780 | hotstack -> 38.102.83.20 | changed: "master"
2025-10-11 12:22:55.987260 | 
2025-10-11 12:22:55.987469 | LOOP [ocp_agent_installer : Generate MachineConfig for iscsi]
2025-10-11 12:22:56.431461 | hotstack -> 38.102.83.20 | changed
2025-10-11 12:22:56.431948 | hotstack -> 38.102.83.20 | changed: All items complete
2025-10-11 12:22:56.432009 | 
2025-10-11 12:22:56.450704 | 
2025-10-11 12:22:56.450953 | LOOP [ocp_agent_installer : Template butane config for multipath]
2025-10-11 12:22:57.355912 | hotstack -> 38.102.83.20 | changed: "master"
2025-10-11 12:22:57.452160 | 
2025-10-11 12:22:57.452370 | LOOP [ocp_agent_installer : Generate MachineConfig for multipath]
2025-10-11 12:22:57.918657 | hotstack -> 38.102.83.20 | changed
2025-10-11 12:22:57.919249 | hotstack -> 38.102.83.20 | changed: All items complete
2025-10-11 12:22:57.919312 | 
2025-10-11 12:22:57.938085 | 
2025-10-11 12:22:57.938168 | LOOP [ocp_agent_installer : Template butane config for LVM cinder-volumes]
2025-10-11 12:22:58.814301 | hotstack -> 38.102.83.20 | changed: "master"
2025-10-11 12:22:58.839156 | 
2025-10-11 12:22:58.839323 | LOOP [ocp_agent_installer : Generate MachineConfig for multipath]
2025-10-11 12:22:59.285461 | hotstack -> 38.102.83.20 | changed
2025-10-11 12:22:59.285908 | hotstack -> 38.102.83.20 | changed: All items complete
2025-10-11 12:22:59.285985 | 
2025-10-11 12:22:59.312370 | 
2025-10-11 12:22:59.312595 | TASK [ocp_agent_installer : Include tasks to generate config assets]
2025-10-11 12:22:59.351325 | hotstack | ok
2025-10-11 12:22:59.370435 | hotstack | included: /var/lib/zuul/builds/2eeebd5c21844fdd9eefbaf9d59e8d1b/trusted/project_3/github.com/openstack-k8s-operators/hotstack/roles/ocp_agent_installer/tasks/config_assets.yml
2025-10-11 12:22:59.400453 | 
2025-10-11 12:22:59.400569 | TASK [ocp_agent_installer : Template ovn-k8s customization]
2025-10-11 12:23:00.286162 | hotstack -> 38.102.83.20 | changed
2025-10-11 12:23:00.300854 | 
2025-10-11 12:23:00.300993 | TASK [ocp_agent_installer : Copy Etcd customization]
2025-10-11 12:23:01.125670 | hotstack -> 38.102.83.20 | changed
2025-10-11 12:23:01.139342 | 
2025-10-11 12:23:01.139488 | TASK [ocp_agent_installer : Template ImageContentSourcePolicy customization]
2025-10-11 12:23:01.208392 | hotstack | skipping: Conditional result was False
2025-10-11 12:23:01.221418 | 
2025-10-11 12:23:01.221565 | TASK [ocp_agent_installer : Initialize _ocp_additional_trusted_ca_map fact]
2025-10-11 12:23:01.269384 | hotstack | skipping: Conditional result was False
2025-10-11 12:23:01.282547 | 
2025-10-11 12:23:01.282705 | LOOP [ocp_agent_installer : Append to _ocp_additional_trusted_ca_map fact]
2025-10-11 12:23:01.330614 | hotstack | skipping: Conditional result was False
2025-10-11 12:23:01.354555 | 
2025-10-11 12:23:01.354812 | TASK [ocp_agent_installer : Template additional CA config map]
2025-10-11 12:23:01.392398 | hotstack | skipping: Conditional result was False
2025-10-11 12:23:01.410552 | 
2025-10-11 12:23:01.410771 | TASK [ocp_agent_installer : Copy additional CA config image]
2025-10-11 12:23:01.448675 | hotstack | skipping: Conditional result was False
2025-10-11 12:23:01.462257 | 
2025-10-11 12:23:01.462453 | TASK [ocp_agent_installer : Copy machine_configs to manifests dir]
2025-10-11 12:23:01.961284 | hotstack -> 38.102.83.20 | changed
2025-10-11 12:23:01.975279 | 
2025-10-11 12:23:01.975428 | TASK [ocp_agent_installer : Copy config assets to manifests dir]
2025-10-11 12:23:02.512986 | hotstack -> 38.102.83.20 | changed
2025-10-11 12:23:02.526503 | 
2025-10-11 12:23:02.526635 | TASK [ocp_agent_installer : Run tasks/pxe_assets.yml]
2025-10-11 12:23:02.574362 | hotstack | ok
2025-10-11 12:23:02.595238 | hotstack | included: /var/lib/zuul/builds/2eeebd5c21844fdd9eefbaf9d59e8d1b/trusted/project_3/github.com/openstack-k8s-operators/hotstack/roles/ocp_agent_installer/tasks/pxe_assets.yml
2025-10-11 12:23:02.628485 | 
2025-10-11 12:23:02.628570 | TASK [ocp_agent_installer : Create the boot-artifacts directory]
2025-10-11 12:23:03.032635 | hotstack -> 38.102.83.20 | changed
2025-10-11 12:23:03.046020 | 
2025-10-11 12:23:03.046156 | TASK [ocp_agent_installer : Create PXE assets]
2025-10-11 12:23:05.053109 | controller-0 | level=warning msg=Compute[0].Hyperthreading: Disabled is ignored
2025-10-11 12:23:05.612672 | controller-0 | level=warning msg=ControlPlane.Hyperthreading: Disabled is ignored
2025-10-11 12:23:05.612709 | controller-0 | level=info msg=Configuration has 3 master replicas and 0 worker replicas
2025-10-11 12:23:05.612721 | controller-0 | level=info msg=The rendezvous host IP (node0 IP) is 192.168.34.10
2025-10-11 12:23:06.412069 | controller-0 | level=info msg=Extracting base ISO from release payload
2025-10-11 12:23:33.137629 | controller-0 | level=info msg=Base ISO obtained from release and cached at [/home/zuul/.cache/agent/image_cache/coreos-x86_64.iso]
2025-10-11 12:24:24.489636 | controller-0 | level=info msg=Consuming Install Config from target directory
2025-10-11 12:24:24.490077 | controller-0 | level=info msg=Consuming Agent Config from target directory
2025-10-11 12:24:25.100554 | controller-0 | level=info msg=Consuming Extra Manifests from target directory
2025-10-11 12:24:25.100612 | controller-0 | level=info msg=Created iPXE script agent.x86_64.ipxe in boot-artifacts directory
2025-10-11 12:24:25.100667 | controller-0 | level=info msg=PXE boot artifacts created in: boot-artifacts
2025-10-11 12:24:25.100704 | controller-0 | level=info msg=Kernel parameters for PXE boot: rw ignition.firstboot ignition.platform.id=metal
2025-10-11 12:24:25.684428 | hotstack -> 38.102.83.20 | changed
2025-10-11 12:24:25.697452 | 
2025-10-11 12:24:25.697596 | TASK [ocp_agent_installer : Set serial console in ipxe]
2025-10-11 12:24:26.396134 | hotstack -> 38.102.83.20 | changed: line replaced
2025-10-11 12:24:26.409368 | 
2025-10-11 12:24:26.409514 | TASK [ocp_agent_installer : Disable net.ifnames]
2025-10-11 12:24:26.998067 | hotstack -> 38.102.83.20 | changed: line replaced
2025-10-11 12:24:27.013071 | 
2025-10-11 12:24:27.013235 | LOOP [ocp_agent_installer : Copy boot-artifacts to the web server - (ocp_agent_installer_boot_artifacts_dir)]
2025-10-11 12:24:28.008921 | hotstack -> 38.102.83.20 | changed: "agent.x86_64-vmlinuz"
2025-10-11 12:24:28.009287 | 
2025-10-11 12:24:29.458449 | hotstack -> 38.102.83.20 | changed: "agent.x86_64-initrd.img"
2025-10-11 12:24:53.323342 | hotstack -> 38.102.83.20 | changed: "agent.x86_64-rootfs.img"
2025-10-11 12:24:53.787048 | hotstack -> 38.102.83.20 | changed: "agent.x86_64.ipxe"
2025-10-11 12:24:53.801156 | 
2025-10-11 12:24:53.801270 | TASK [ocp_agent_installer : Run tasks/iso_assets.yml]
2025-10-11 12:24:53.837390 | hotstack | skipping: Conditional result was False
2025-10-11 12:24:53.849715 | 
2025-10-11 12:24:53.849840 | TASK [ocp_agent_installer : Copy auth/kubeconfig to ~/.kube/config]
2025-10-11 12:24:54.394675 | hotstack -> 38.102.83.20 | changed
2025-10-11 12:24:54.408253 | 
2025-10-11 12:24:54.408402 | TASK [ocp_agent_installer : Wait for bootstrap-complete]
2025-10-11 12:25:35.021177 | controller-0 | level=info msg=Waiting for cluster install to initialize. Sleeping for 30 seconds
2025-10-11 12:26:17.043515 | controller-0 | level=info msg=Waiting for cluster install to initialize. Sleeping for 30 seconds
2025-10-11 12:26:59.070270 | controller-0 | level=info msg=Cannot access Rendezvous Host. There may be a network configuration problem, check console for additional info
2025-10-11 12:27:41.091754 | controller-0 | level=info msg=Cannot access Rendezvous Host. There may be a network configuration problem, check console for additional info
2025-10-11 12:28:23.136792 | controller-0 | level=info msg=Cluster is not ready for install. Check validations
2025-10-11 12:28:23.137295 | controller-0 | level=warning msg=Cluster validation: The cluster has hosts that are not ready to install.
2025-10-11 12:28:23.138960 | controller-0 | level=warning msg=Host master-0 validation: No connectivity to the majority of hosts in the cluster
2025-10-11 12:28:23.139060 | controller-0 | level=warning msg=Host master-0 validation: Host couldn't synchronize with any NTP server
2025-10-11 12:28:23.139252 | controller-0 | level=warning msg=Host master-0 validation: Error while evaluating DNS resolution on this host
2025-10-11 12:28:23.139321 | controller-0 | level=warning msg=Host master-0 validation: Error while evaluating DNS resolution on this host
2025-10-11 12:28:23.139426 | controller-0 | level=warning msg=Host master-0 validation: Error while evaluating DNS resolution on this host
2025-10-11 12:28:23.140402 | controller-0 | level=warning msg=Host master-1 validation: No connectivity to the majority of hosts in the cluster
2025-10-11 12:28:23.140501 | controller-0 | level=warning msg=Host master-1 validation: Host couldn't synchronize with any NTP server
2025-10-11 12:28:23.140656 | controller-0 | level=warning msg=Host master-1 validation: Error while evaluating DNS resolution on this host
2025-10-11 12:28:23.140723 | controller-0 | level=warning msg=Host master-1 validation: Error while evaluating DNS resolution on this host
2025-10-11 12:28:23.140788 | controller-0 | level=warning msg=Host master-1 validation: Error while evaluating DNS resolution on this host
2025-10-11 12:28:23.141677 | controller-0 | level=warning msg=Host master-2 validation: No connectivity to the majority of hosts in the cluster
2025-10-11 12:28:23.141774 | controller-0 | level=warning msg=Host master-2 validation: Host couldn't synchronize with any NTP server
2025-10-11 12:28:23.141961 | controller-0 | level=warning msg=Host master-2 validation: Error while evaluating DNS resolution on this host
2025-10-11 12:28:23.142037 | controller-0 | level=warning msg=Host master-2 validation: Error while evaluating DNS resolution on this host
2025-10-11 12:28:23.142102 | controller-0 | level=warning msg=Host master-2 validation: Error while evaluating DNS resolution on this host
2025-10-11 12:28:23.148909 | controller-0 | level=warning msg=Host master-2: validation 'belongs-to-majority-group' failed
2025-10-11 12:29:23.361717 | controller-0 | level=info msg=Host master-1 validation: Host NTP is synced
2025-10-11 12:29:23.361842 | controller-0 | level=info msg=Host master-1 validation: Domain name resolution for the api.ocp.openstack.lab domain was successful or not required
2025-10-11 12:29:23.361909 | controller-0 | level=info msg=Host master-1 validation: Domain name resolution for the api-int.ocp.openstack.lab domain was successful or not required
2025-10-11 12:29:23.361961 | controller-0 | level=info msg=Host master-1 validation: Domain name resolution for the *.apps.ocp.openstack.lab domain was successful or not required
2025-10-11 12:29:23.362171 | controller-0 | level=info msg=Host master-2 validation: Host NTP is synced
2025-10-11 12:29:23.362235 | controller-0 | level=info msg=Host master-2 validation: Domain name resolution for the api.ocp.openstack.lab domain was successful or not required
2025-10-11 12:29:23.362274 | controller-0 | level=info msg=Host master-2 validation: Domain name resolution for the api-int.ocp.openstack.lab domain was successful or not required
2025-10-11 12:29:23.362344 | controller-0 | level=info msg=Host master-2 validation: Domain name resolution for the *.apps.ocp.openstack.lab domain was successful or not required
2025-10-11 12:29:23.362555 | controller-0 | level=info msg=Host master-0 validation: Host NTP is synced
2025-10-11 12:29:23.362619 | controller-0 | level=info msg=Host master-0 validation: Domain name resolution for the api.ocp.openstack.lab domain was successful or not required
2025-10-11 12:29:23.362694 | controller-0 | level=info msg=Host master-0 validation: Domain name resolution for the api-int.ocp.openstack.lab domain was successful or not required
2025-10-11 12:29:23.362742 | controller-0 | level=info msg=Host master-0 validation: Domain name resolution for the *.apps.ocp.openstack.lab domain was successful or not required
2025-10-11 12:29:23.366541 | controller-0 | level=info msg=Host master-0: validation 'ntp-synced' is now fixed
2025-10-11 12:29:35.406567 | controller-0 | level=info msg=Host master-2 validation: Host has connectivity to the majority of hosts in the cluster
2025-10-11 12:29:35.407078 | controller-0 | level=info msg=Host master-1 validation: Host has connectivity to the majority of hosts in the cluster
2025-10-11 12:29:35.407539 | controller-0 | level=info msg=Host master-0 validation: Host has connectivity to the majority of hosts in the cluster
2025-10-11 12:29:35.411767 | controller-0 | level=info msg=Host master-0: updated status from insufficient to known (Host is ready to be installed)
2025-10-11 12:29:47.453506 | controller-0 | level=info msg=Preparing cluster for installation
2025-10-11 12:29:47.453860 | controller-0 | level=info msg=Cluster validation: All hosts in the cluster are ready to install.
2025-10-11 12:29:59.506302 | controller-0 | level=info msg=Host master-0: updated status from known to preparing-for-installation (Host finished successfully to prepare for installation)
2025-10-11 12:30:35.643886 | controller-0 | level=info msg=Host master-1: updated status from preparing-for-installation to preparing-successful (Host finished successfully to prepare for installation)
2025-10-11 12:32:36.100164 | controller-0 | level=warning msg=Host's disk /dev/disk/by-path/pci-0000:00:06.0 is slower than the supported speed, and may cause degraded cluster performance (fdatasync duration: 90 ms)
2025-10-11 12:32:48.142260 | controller-0 | level=info msg=Cluster installation in progress
2025-10-11 12:32:48.149117 | controller-0 | level=info msg=Host master-2: updated status from preparing-for-installation to preparing-successful (Host finished successfully to prepare for installation)
2025-10-11 12:33:00.201120 | controller-0 | level=info msg=Host master-2: updated status from preparing-successful to installing (Installation is in progress)
2025-10-11 12:33:24.309833 | controller-0 | level=info msg=Host: master-0, reached installation stage Writing image to disk: 19%
2025-10-11 12:33:36.358624 | controller-0 | level=info msg=Host: master-2, reached installation stage Writing image to disk: 54%
2025-10-11 12:33:48.406468 | controller-0 | level=info msg=Host: master-2, reached installation stage Writing image to disk: 89%
2025-10-11 12:34:00.459909 | controller-0 | level=info msg=Host: master-0, reached installation stage Writing image to disk: 100%
2025-10-11 12:34:35.585287 | controller-0 | level=info msg=Bootstrap Kube API Initialized
2025-10-11 12:35:01.983006 | controller-0 | level=info msg=Host: master-1, reached installation stage Rebooting
2025-10-11 12:38:47.409214 | controller-0 | level=info msg=Host: master-1, reached installation stage Joined
2025-10-11 12:38:55.533300 | controller-0 | level=info msg=Host: master-1, reached installation stage Done
2025-10-11 12:39:05.686907 | controller-0 | level=info msg=Node master-1 has been rebooted 1 times before completing installation
2025-10-11 13:05:04.820776 | controller-0 | level=warning msg=Host master-2: updated status from installing-in-progress to error (Host failed to install because its installation stage Writing image to disk did not sufficiently progress in the last 30m0s.)
2025-10-11 13:05:08.878065 | controller-0 | level=info msg=Cluster has hosts in error
2025-10-11 13:05:08.878244 | controller-0 | level=info msg=cluster has stopped installing... working to recover installation
2025-10-11 13:24:55.020274 | controller-0 | level=error msg=Cluster operator authentication Degraded is True with IngressStateEndpoints_MissingSubsets::OAuthServerServiceEndpointAccessibleController_SyncError::OAuthServerServiceEndpointsEndpointAccessibleController_SyncError: IngressStateEndpointsDegraded: No subsets found for the endpoints of oauth-server
2025-10-11 13:24:55.020445 | controller-0 | level=error msg=OAuthServerServiceEndpointAccessibleControllerDegraded: Get "https://172.30.101.250:443/healthz": dial tcp 172.30.101.250:443: connect: connection refused
2025-10-11 13:24:55.020494 | controller-0 | level=error msg=OAuthServerServiceEndpointsEndpointAccessibleControllerDegraded: oauth service endpoints are not ready
2025-10-11 13:24:55.020628 | controller-0 | level=info msg=Cluster operator authentication Available is False with APIServices_PreconditionNotReady::OAuthServerServiceEndpointAccessibleController_EndpointUnavailable::OAuthServerServiceEndpointsEndpointAccessibleController_ResourceNotFound: APIServicesAvailable: PreconditionNotReady
2025-10-11 13:24:55.020676 | controller-0 | level=info msg=OAuthServerServiceEndpointAccessibleControllerAvailable: Get "https://172.30.101.250:443/healthz": dial tcp 172.30.101.250:443: connect: connection refused
2025-10-11 13:24:55.020740 | controller-0 | level=info msg=OAuthServerServiceEndpointsEndpointAccessibleControllerAvailable: endpoints "oauth-openshift" not found
2025-10-11 13:24:55.020831 | controller-0 | level=info msg=Cluster operator authentication EvaluationConditionsDetected is Unknown with NoData:
2025-10-11 13:24:55.020899 | controller-0 | level=info msg=Cluster operator baremetal Disabled is False with :
2025-10-11 13:24:55.020965 | controller-0 | level=info msg=Cluster operator cloud-controller-manager TrustedCABundleControllerControllerAvailable is True with AsExpected: Trusted CA Bundle Controller works as expected
2025-10-11 13:24:55.021035 | controller-0 | level=info msg=Cluster operator cloud-controller-manager TrustedCABundleControllerControllerDegraded is False with AsExpected: Trusted CA Bundle Controller works as expected
2025-10-11 13:24:55.021106 | controller-0 | level=info msg=Cluster operator cloud-controller-manager CloudConfigControllerAvailable is True with AsExpected: Cloud Config Controller works as expected
2025-10-11 13:24:55.021176 | controller-0 | level=info msg=Cluster operator cloud-controller-manager CloudConfigControllerDegraded is False with AsExpected: Cloud Config Controller works as expected
2025-10-11 13:24:55.021253 | controller-0 | level=info msg=Cluster operator config-operator EvaluationConditionsDetected is Unknown with NoData:
2025-10-11 13:24:55.021321 | controller-0 | level=info msg=Cluster operator csi-snapshot-controller EvaluationConditionsDetected is Unknown with NoData:
2025-10-11 13:24:55.021438 | controller-0 | level=error msg=Cluster operator etcd Degraded is True with RevisionController_SyncError: RevisionControllerDegraded: etcd cluster has quorum of 2 which is not fault tolerant: [{Member:ID:13113484225824453998 name:"master-1" peerURLs:"https://192.168.34.11:2380" clientURLs:"https://192.168.34.11:2379"  Healthy:true Took:6.760742ms Error:<nil>} {Member:ID:14813819669272913224 name:"etcd-bootstrap" peerURLs:"https://192.168.34.10:2380" clientURLs:"https://192.168.34.10:2379"  Healthy:true Took:1.877713ms Error:<nil>}]
2025-10-11 13:24:55.021524 | controller-0 | level=info msg=Cluster operator etcd EvaluationConditionsDetected is Unknown with NoData:
2025-10-11 13:24:55.021620 | controller-0 | level=info msg=Cluster operator ingress Available is False with IngressUnavailable: The "default" ingress controller reports Available=False: IngressControllerUnavailable: One or more status conditions indicate unavailable: DeploymentAvailable=False (DeploymentUnavailable: The deployment has Available status condition set to False (reason: MinimumReplicasUnavailable) with message: Deployment does not have minimum availability.)
2025-10-11 13:24:55.021719 | controller-0 | level=info msg=Cluster operator ingress Progressing is True with Reconciling: ingresscontroller "default" is progressing: IngressControllerProgressing: One or more status conditions indicate progressing: DeploymentRollingOut=True (DeploymentRollingOut: Waiting for router deployment rollout to finish: 0 of 2 updated replica(s) are available...
2025-10-11 13:24:55.021763 | controller-0 | level=info msg=).
2025-10-11 13:24:55.021803 | controller-0 | level=info msg=Not all ingress controllers are available.
2025-10-11 13:24:55.021888 | controller-0 | level=error msg=Cluster operator ingress Degraded is True with IngressDegraded: The "default" ingress controller reports Degraded=True: DegradedConditions: One or more other status conditions indicate a degraded state: DeploymentAvailable=False (DeploymentUnavailable: The deployment has Available status condition set to False (reason: MinimumReplicasUnavailable) with message: Deployment does not have minimum availability.), DeploymentReplicasMinAvailable=False (DeploymentMinimumReplicasNotMet: 0/2 of replicas are available, max unavailable is 1: Some pods are not scheduled: Pod "router-default-5ddb89f76-whh4f" cannot be scheduled: 0/1 nodes are available: 1 node(s) didn't have free ports for the requested pod ports. preemption: 0/1 nodes are available: 1 node(s) didn't have free ports for the requested pod ports. Make sure you have sufficient worker nodes.)
2025-10-11 13:24:55.021998 | controller-0 | level=info msg=Cluster operator ingress EvaluationConditionsDetected is False with AsExpected:
2025-10-11 13:24:55.022069 | controller-0 | level=info msg=Cluster operator insights ClusterTransferAvailable is False with NoClusterTransfer: no available cluster transfer
2025-10-11 13:24:55.022193 | controller-0 | level=info msg=Cluster operator insights Disabled is False with AsExpected:
2025-10-11 13:24:55.022262 | controller-0 | level=info msg=Cluster operator insights RemoteConfigurationAvailable is True with AsExpected:
2025-10-11 13:24:55.022353 | controller-0 | level=info msg=Cluster operator insights RemoteConfigurationValid is True with AsExpected:
2025-10-11 13:24:55.022436 | controller-0 | level=info msg=Cluster operator insights SCAAvailable is True with Updated: SCA certs successfully updated in the etc-pki-entitlement secret
2025-10-11 13:24:55.022505 | controller-0 | level=info msg=Cluster operator kube-apiserver EvaluationConditionsDetected is False with AsExpected: All is well
2025-10-11 13:24:55.022570 | controller-0 | level=info msg=Cluster operator kube-controller-manager EvaluationConditionsDetected is Unknown with NoData:
2025-10-11 13:24:55.022634 | controller-0 | level=info msg=Cluster operator kube-scheduler EvaluationConditionsDetected is Unknown with NoData:
2025-10-11 13:24:55.022710 | controller-0 | level=info msg=Cluster operator kube-storage-version-migrator EvaluationConditionsDetected is Unknown with NoData:
2025-10-11 13:24:55.022783 | controller-0 | level=info msg=Cluster operator machine-config EvaluationConditionsDetected is False with AsExpected:
2025-10-11 13:24:55.022850 | controller-0 | level=info msg=Cluster operator monitoring Available is False with UpdatingPrometheusOperatorFailed: UpdatingPrometheusOperator: reconciling Prometheus Operator Admission Webhook Deployment failed: updating Deployment object failed: waiting for DeploymentRollout of openshift-monitoring/prometheus-operator-admission-webhook: context deadline exceeded: got 1 unavailable replicas
2025-10-11 13:24:55.022933 | controller-0 | level=error msg=Cluster operator monitoring Degraded is True with UpdatingPrometheusOperatorFailed: UpdatingPrometheusOperator: reconciling Prometheus Operator Admission Webhook Deployment failed: updating Deployment object failed: waiting for DeploymentRollout of openshift-monitoring/prometheus-operator-admission-webhook: context deadline exceeded: got 1 unavailable replicas
2025-10-11 13:24:55.023010 | controller-0 | level=info msg=Cluster operator monitoring Progressing is True with RollOutInProgress: Rolling out the stack.
2025-10-11 13:24:55.023076 | controller-0 | level=info msg=Cluster operator network ManagementStateDegraded is False with :
2025-10-11 13:24:55.023144 | controller-0 | level=info msg=Cluster operator network Progressing is True with Deploying: Deployment "/openshift-ovn-kubernetes/ovnkube-control-plane" is not available (awaiting 1 nodes)
2025-10-11 13:24:55.023215 | controller-0 | level=info msg=Cluster operator network Available is False with Startup: The network is starting up
2025-10-11 13:24:55.023288 | controller-0 | level=error msg=Cluster operator olm Degraded is True with CatalogdClusterCatalogOpenshiftCertifiedOperators_SyncError::CatalogdClusterCatalogOpenshiftCommunityOperators_SyncError::CatalogdClusterCatalogOpenshiftRedhatMarketplace_SyncError::CatalogdClusterCatalogOpenshiftRedhatOperators_SyncError: CatalogdClusterCatalogOpenshiftCertifiedOperatorsDegraded: Internal error occurred: failed calling webhook "inject-metadata-name.olm.operatorframework.io": failed to call webhook: Post "https://catalogd-service.openshift-catalogd.svc:9443/mutate-olm-operatorframework-io-v1-clustercatalog?timeout=10s": no endpoints available for service "catalogd-service"
2025-10-11 13:24:55.023361 | controller-0 | level=error msg=CatalogdClusterCatalogOpenshiftCommunityOperatorsDegraded: Internal error occurred: failed calling webhook "inject-metadata-name.olm.operatorframework.io": failed to call webhook: Post "https://catalogd-service.openshift-catalogd.svc:9443/mutate-olm-operatorframework-io-v1-clustercatalog?timeout=10s": no endpoints available for service "catalogd-service"
2025-10-11 13:24:55.023426 | controller-0 | level=error msg=CatalogdClusterCatalogOpenshiftRedhatMarketplaceDegraded: Internal error occurred: failed calling webhook "inject-metadata-name.olm.operatorframework.io": failed to call webhook: Post "https://catalogd-service.openshift-catalogd.svc:9443/mutate-olm-operatorframework-io-v1-clustercatalog?timeout=10s": no endpoints available for service "catalogd-service"
2025-10-11 13:24:55.023469 | controller-0 | level=error msg=CatalogdClusterCatalogOpenshiftRedhatOperatorsDegraded: Internal error occurred: failed calling webhook "inject-metadata-name.olm.operatorframework.io": failed to call webhook: Post "https://catalogd-service.openshift-catalogd.svc:9443/mutate-olm-operatorframework-io-v1-clustercatalog?timeout=10s": no endpoints available for service "catalogd-service"
2025-10-11 13:24:55.023615 | controller-0 | level=info msg=Cluster operator olm Progressing is True with CatalogdDeploymentCatalogdControllerManager_Deploying::OperatorcontrollerDeploymentOperatorControllerControllerManager_Deploying: CatalogdDeploymentCatalogdControllerManagerProgressing: Waiting for Deployment to deploy pods
2025-10-11 13:24:55.023658 | controller-0 | level=info msg=OperatorcontrollerDeploymentOperatorControllerControllerManagerProgressing: Waiting for Deployment to deploy pods
2025-10-11 13:24:55.023741 | controller-0 | level=info msg=Cluster operator olm Available is False with CatalogdDeploymentCatalogdControllerManager_Deploying::OperatorcontrollerDeploymentOperatorControllerControllerManager_Deploying: CatalogdDeploymentCatalogdControllerManagerAvailable: Waiting for Deployment
2025-10-11 13:24:55.023782 | controller-0 | level=info msg=OperatorcontrollerDeploymentOperatorControllerControllerManagerAvailable: Waiting for Deployment
2025-10-11 13:24:55.023879 | controller-0 | level=info msg=Cluster operator olm EvaluationConditionsDetected is Unknown with NoData:
2025-10-11 13:24:55.023948 | controller-0 | level=info msg=Cluster operator openshift-apiserver Available is False with APIServices_PreconditionNotReady: APIServicesAvailable: PreconditionNotReady
2025-10-11 13:24:55.024016 | controller-0 | level=info msg=Cluster operator openshift-apiserver EvaluationConditionsDetected is Unknown with NoData:
2025-10-11 13:24:55.024084 | controller-0 | level=info msg=Cluster operator openshift-controller-manager EvaluationConditionsDetected is Unknown with NoData:
2025-10-11 13:24:55.024177 | controller-0 | level=info msg=Cluster operator operator-lifecycle-manager-packageserver Available is False with ClusterServiceVersionNotSucceeded: ClusterServiceVersion openshift-operator-lifecycle-manager/packageserver observed in phase Failed with reason: InstallCheckFailed, message: install failed: deployment packageserver not ready before timeout: deployment "packageserver" exceeded its progress deadline
2025-10-11 13:24:55.024258 | controller-0 | level=info msg=Cluster operator service-ca EvaluationConditionsDetected is Unknown with NoData:
2025-10-11 13:24:55.024325 | controller-0 | level=info msg=Cluster operator storage EvaluationConditionsDetected is Unknown with NoData:
2025-10-11 13:24:55.024433 | controller-0 | level=info msg=Use the following commands to gather logs from the cluster
2025-10-11 13:24:55.024495 | controller-0 | level=info msg=openshift-install gather bootstrap --help
2025-10-11 13:24:55.024581 | controller-0 | level=error msg=Bootstrap failed to complete: : bootstrap process timed out: context deadline exceeded
2025-10-11 13:24:55.421200 | hotstack -> 38.102.83.20 | ERROR
2025-10-11 13:24:55.421468 | hotstack -> 38.102.83.20 | {
2025-10-11 13:24:55.421528 | hotstack -> 38.102.83.20 |   "delta": "1:00:00.207694",
2025-10-11 13:24:55.421570 | hotstack -> 38.102.83.20 |   "end": "2025-10-11 13:24:55.038722",
2025-10-11 13:24:55.421608 | hotstack -> 38.102.83.20 |   "msg": "non-zero return code",
2025-10-11 13:24:55.421645 | hotstack -> 38.102.83.20 |   "rc": 5,
2025-10-11 13:24:55.421681 | hotstack -> 38.102.83.20 |   "start": "2025-10-11 12:24:54.831028"
2025-10-11 13:24:55.421716 | hotstack -> 38.102.83.20 | }
failure
2025-10-11 13:24:55.426036 | 
2025-10-11 13:24:55.426133 | PLAY RECAP
2025-10-11 13:24:55.426211 | hotstack | ok: 74 changed: 48 unreachable: 0 failed: 1 skipped: 12 rescued: 0 ignored: 0
2025-10-11 13:24:55.426257 | 
2025-10-11 13:24:55.567313 | PRE-RUN END RESULT_NORMAL: [trusted : review.rdoproject.org/config/playbooks/hotstack/pre_stage2.yaml@master]
2025-10-11 13:24:55.570022 | POST-RUN START: [trusted : review.rdoproject.org/config/playbooks/hotstack/post.yaml@master]
2025-10-11 13:24:56.109414 | 
2025-10-11 13:24:56.109520 | PLAY [Collect hotstack logs and clean up environment]
2025-10-11 13:24:56.130349 | 
2025-10-11 13:24:56.130426 | TASK [Fetch stack outputs from remote to zuul executor]
2025-10-11 13:24:56.789234 | hotstack | Output suppressed because no_log was given
2025-10-11 13:24:56.800228 | 
2025-10-11 13:24:56.800378 | TASK [Load stack output vars with template resolution]
2025-10-11 13:24:56.847300 | hotstack | Output suppressed because no_log was given
2025-10-11 13:24:56.858099 | 
2025-10-11 13:24:56.858234 | TASK [Add Hotstack controller-0 to ansible inventory]
2025-10-11 13:24:56.895363 | hotstack | Output suppressed because no_log was given
2025-10-11 13:24:56.907057 | 
2025-10-11 13:24:56.907193 | TASK [Wait for Hotstack controller-0 to be ready]
2025-10-11 13:25:02.781614 | hotstack -> 38.102.83.20 | Output suppressed because no_log was given
2025-10-11 13:25:02.796320 | 
2025-10-11 13:25:02.796486 | TASK [Collect hotstack logs]
2025-10-11 13:25:02.824666 | hotstack | Output suppressed because no_log was given
2025-10-11 13:25:02.866575 | 
2025-10-11 13:25:02.866715 | TASK [hotlogs : Initialize failure tracking]
2025-10-11 13:25:02.898313 | hotstack | ok
2025-10-11 13:25:02.905146 | 
2025-10-11 13:25:02.905230 | TASK [hotlogs : Ensure logs directory]
2025-10-11 13:25:03.329019 | hotstack | changed
2025-10-11 13:25:03.340088 | 
2025-10-11 13:25:03.340226 | LOOP [hotlogs : Create hotlogs sub dirs]
2025-10-11 13:25:03.646301 | hotstack | changed:
2025-10-11 13:25:03.646849 | hotstack | {
2025-10-11 13:25:03.646917 | hotstack |   "dest_dir": "ocp_cluster",
2025-10-11 13:25:03.646960 | hotstack |   "src": "/home/zuul/ocp-cluster/.openshift_install.log"
2025-10-11 13:25:03.647032 | hotstack | }
2025-10-11 13:25:03.887778 | hotstack | ok:
2025-10-11 13:25:03.887930 | hotstack | {
2025-10-11 13:25:03.887980 | hotstack |   "dest_dir": "ocp_cluster",
2025-10-11 13:25:03.888021 | hotstack |   "src": "/home/zuul/cluster-custom-config/"
2025-10-11 13:25:03.888064 | hotstack | }
2025-10-11 13:25:04.145299 | hotstack | changed:
2025-10-11 13:25:04.145459 | hotstack | {
2025-10-11 13:25:04.145509 | hotstack |   "dest_dir": "data",
2025-10-11 13:25:04.145548 | hotstack |   "src": "/home/zuul/data/"
2025-10-11 13:25:04.145586 | hotstack | }
2025-10-11 13:25:04.403037 | hotstack | changed:
2025-10-11 13:25:04.403208 | hotstack | {
2025-10-11 13:25:04.403259 | hotstack |   "dest_dir": "manifests",
2025-10-11 13:25:04.403300 | hotstack |   "src": "/home/zuul/manifests/"
2025-10-11 13:25:04.403337 | hotstack | }
2025-10-11 13:25:04.646953 | hotstack | ok:
2025-10-11 13:25:04.647120 | hotstack | {
2025-10-11 13:25:04.647169 | hotstack |   "dest_dir": ".",
2025-10-11 13:25:04.647208 | hotstack |   "src": "/home/zuul/must-gather.tar.gz"
2025-10-11 13:25:04.647244 | hotstack | }
2025-10-11 13:25:04.672253 | 
2025-10-11 13:25:04.672519 | TASK [hotlogs : Run must-gather operations]
2025-10-11 13:25:05.594350 | hotstack -> 38.102.83.20 | ERROR
2025-10-11 13:25:05.594856 | hotstack -> 38.102.83.20 | {
2025-10-11 13:25:05.594939 | hotstack -> 38.102.83.20 |   "msg": "Must-gather failed: error: unable to resolve image stream 'openshift/must-gather': the server could not find the requested resource (get imagestreams.image.openshift.io must-gather)\n (Command: oc adm must-gather --image-stream=openshift/must-gather --image=quay.io/openstack-k8s-operators/openstack-must-gather --dest-dir=/home/zuul/must-gather --timeout=10m --host-network=false -- ADDITIONAL_NAMESPACES=sushy-emulator OPENSTACK_DATABASES=ALL SOS_EDPM=all SOS_DECOMPRESS=0 gather)",
2025-10-11 13:25:05.594993 | hotstack -> 38.102.83.20 |   "rc": 1
2025-10-11 13:25:05.595032 | hotstack -> 38.102.83.20 | }
2025-10-11 13:25:05.608539 | 
2025-10-11 13:25:05.608681 | TASK [hotlogs : Display must-gather failure]
2025-10-11 13:25:05.752506 | hotstack | ok:
2025-10-11 13:25:05.752870 | hotstack | Must-gather operations failed: Must-gather failed: error: unable to resolve image stream 'openshift/must-gather': the server could not find the requested resource (get imagestreams.image.openshift.io must-gather)
2025-10-11 13:25:05.752966 | hotstack |  (Command: oc adm must-gather --image-stream=openshift/must-gather --image=quay.io/openstack-k8s-operators/openstack-must-gather --dest-dir=/home/zuul/must-gather --timeout=10m --host-network=false -- ADDITIONAL_NAMESPACES=sushy-emulator OPENSTACK_DATABASES=ALL SOS_EDPM=all SOS_DECOMPRESS=0 gather)
2025-10-11 13:25:05.804590 | 
2025-10-11 13:25:05.804769 | TASK [hotlogs : Track must-gather failure]
2025-10-11 13:25:05.863083 | hotstack | ok
2025-10-11 13:25:05.877080 | 
2025-10-11 13:25:05.877244 | LOOP [hotlogs : Collect hotstack data]
2025-10-11 13:25:06.759237 | hotstack -> 38.102.83.97 | changed: >f+++++++++ .openshift_install.log
2025-10-11 13:25:06.759642 | hotstack -> 38.102.83.97 | changed: All items complete
2025-10-11 13:25:06.759701 | 
2025-10-11 13:25:07.381344 | hotstack -> 38.102.83.97 | changed:
2025-10-11 13:25:07.381505 | hotstack -> 38.102.83.97 | .d..t...... ./
2025-10-11 13:25:07.381558 | hotstack -> 38.102.83.97 | cd+++++++++ butane/
2025-10-11 13:25:07.381601 | hotstack -> 38.102.83.97 | >f+++++++++ butane/90-master-disable-netifnames.bu
2025-10-11 13:25:07.381640 | hotstack -> 38.102.83.97 | >f+++++++++ butane/90-master-enable-iscsi.bu
2025-10-11 13:25:07.381678 | hotstack -> 38.102.83.97 | >f+++++++++ butane/91-master-enable-multipath.bu
2025-10-11 13:25:07.381716 | hotstack -> 38.102.83.97 | >f+++++++++ butane/92-master-lv-cinder-volumes.bu
2025-10-11 13:25:07.381791 | hotstack -> 38.102.83.97 | cd+++++++++ config-assets/
2025-10-11 13:25:07.381833 | hotstack -> 38.102.83.97 | >f+++++++++ config-assets/95-etcd_config.yaml
2025-10-11 13:25:07.381871 | hotstack -> 38.102.83.97 | >f+++++++++ config-assets/ovn_k8s_config.yaml
2025-10-11 13:25:07.381907 | hotstack -> 38.102.83.97 | cd+++++++++ machine-configs/
2025-10-11 13:25:07.381945 | hotstack -> 38.102.83.97 | >f+++++++++ machine-configs/90-master-disable-netifnames.yaml
2025-10-11 13:25:07.381982 | hotstack -> 38.102.83.97 | >f+++++++++ machine-configs/90-master-enable-iscsi.yaml
2025-10-11 13:25:07.382019 | hotstack -> 38.102.83.97 | >f+++++++++ machine-configs/91-master-enable-multipath.yaml
2025-10-11 13:25:07.382056 | hotstack -> 38.102.83.97 | >f+++++++++ machine-configs/92-master-lv-cinder-volumes.yaml
2025-10-11 13:25:07.952869 | hotstack -> 38.102.83.97 | changed: .d..t...... ./
2025-10-11 13:25:08.616642 | hotstack -> 38.102.83.97 | ERROR: Item: {'src': '/home/zuul/manifests/', 'dest_dir': 'manifests'}
2025-10-11 13:25:08.616882 | hotstack -> 38.102.83.97 | {
2025-10-11 13:25:08.616927 | hotstack -> 38.102.83.97 |   "ansible_loop_var": "item",
2025-10-11 13:25:08.616954 | hotstack -> 38.102.83.97 |   "item": {
2025-10-11 13:25:08.616979 | hotstack -> 38.102.83.97 |     "dest_dir": "manifests",
2025-10-11 13:25:08.617003 | hotstack -> 38.102.83.97 |     "src": "/home/zuul/manifests/"
2025-10-11 13:25:08.617027 | hotstack -> 38.102.83.97 |   },
2025-10-11 13:25:08.617051 | hotstack -> 38.102.83.97 |   "msg": "Warning: Permanently added '38.102.83.20' (ED25519) to the list of known hosts.\r\nrsync: [sender] change_dir \"/home/zuul/manifests\" failed: No such file or directory (2)\nrsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1839) [Receiver=3.2.5]\nrsync: [Receiver] write error: Broken pipe (32)\n",
2025-10-11 13:25:08.617078 | hotstack -> 38.102.83.97 |   "rc": 23
2025-10-11 13:25:08.617102 | hotstack -> 38.102.83.97 | }
2025-10-11 13:25:09.315361 | hotstack -> 38.102.83.97 | ERROR: Item: {'src': '/home/zuul/must-gather.tar.gz', 'dest_dir': '.'}
2025-10-11 13:25:09.315576 | hotstack -> 38.102.83.97 | {
2025-10-11 13:25:09.315637 | hotstack -> 38.102.83.97 |   "ansible_loop_var": "item",
2025-10-11 13:25:09.315679 | hotstack -> 38.102.83.97 |   "item": {
2025-10-11 13:25:09.315719 | hotstack -> 38.102.83.97 |     "dest_dir": ".",
2025-10-11 13:25:09.315864 | hotstack -> 38.102.83.97 |     "src": "/home/zuul/must-gather.tar.gz"
2025-10-11 13:25:09.315904 | hotstack -> 38.102.83.97 |   },
2025-10-11 13:25:09.315940 | hotstack -> 38.102.83.97 |   "msg": "Warning: Permanently added '38.102.83.20' (ED25519) to the list of known hosts.\r\nrsync: [sender] link_stat \"/home/zuul/must-gather.tar.gz\" failed: No such file or directory (2)\nrsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1839) [Receiver=3.2.5]\nrsync: [Receiver] write error: Broken pipe (32)\n",
2025-10-11 13:25:09.315999 | hotstack -> 38.102.83.97 |   "rc": 23
2025-10-11 13:25:09.316055 | hotstack -> 38.102.83.97 | }
2025-10-11 13:25:09.332082 | hotstack | ERROR: Ignoring Errors
2025-10-11 13:25:09.346422 | 
2025-10-11 13:25:09.346624 | LOOP [hotlogs : Check hotstack data collection failures]
2025-10-11 13:25:09.397646 | hotstack | skipping: Conditional result was False
2025-10-11 13:25:09.403973 | hotstack | skipping: Conditional result was False
2025-10-11 13:25:09.410973 | hotstack | skipping: Conditional result was False
2025-10-11 13:25:09.427573 | hotstack | ERROR
2025-10-11 13:25:09.427810 | hotstack | {
2025-10-11 13:25:09.427875 | hotstack |   "msg": "Unexpected templating type error occurred on ({{ hotlogs_failures + ['Failed to collect: ' + item.item] }}): can only concatenate str (not \"dict\") to str. can only concatenate str (not \"dict\") to str"
2025-10-11 13:25:09.427919 | hotstack | }
2025-10-11 13:25:09.437999 | 
2025-10-11 13:25:09.438167 | TASK [Print message that log collection failed]
2025-10-11 13:25:09.475560 | hotstack | ok: WARNING: Log collection failed and may be incomplete. Continue with cleanup ...
2025-10-11 13:25:09.485919 | 
2025-10-11 13:25:09.486103 | TASK [Set _auth_type fact]
2025-10-11 13:25:09.532511 | hotstack | Output suppressed because no_log was given
2025-10-11 13:25:09.543223 | 
2025-10-11 13:25:09.543366 | TASK [Write clouds.yaml]
2025-10-11 13:25:10.582580 | hotstack | Output suppressed because no_log was given
2025-10-11 13:25:10.593442 | 
2025-10-11 13:25:10.593535 | TASK [Delete application credentials]
2025-10-11 13:25:12.632098 | hotstack | changed
2025-10-11 13:25:12.644002 | 
2025-10-11 13:25:12.644139 | TASK [Print message that application credential failed to delete]
2025-10-11 13:25:12.671344 | hotstack | skipping: Conditional result was False
2025-10-11 13:25:12.683066 | 
2025-10-11 13:25:12.683202 | TASK [Print message that application credential deleted successfully]
2025-10-11 13:25:12.729788 | hotstack | ok: Application credential deleted successfully.
2025-10-11 13:25:12.740899 | 
2025-10-11 13:25:12.741028 | TASK [Write clouds.yaml with application credentials]
2025-10-11 13:25:12.777929 | 
2025-10-11 13:25:12.778174 | TASK [Delete the heat stack]
2025-10-11 09:25:45.481361 | hotstack | 2025-10-11 13:25:15Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b]: DELETE_IN_PROGRESS  Stack DELETE started
2025-10-11 09:25:45.481491 | hotstack | 2025-10-11 13:25:15Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.controller-floating-ip]: DELETE_IN_PROGRESS  state changed
2025-10-11 09:25:45.481510 | hotstack | 2025-10-11 13:25:16Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.controller-floating-ip]: DELETE_COMPLETE  state changed
2025-10-11 09:25:45.481524 | hotstack | 2025-10-11 13:25:16Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.machine-net-router-interface]: DELETE_IN_PROGRESS  state changed
2025-10-11 09:25:45.481538 | hotstack | 2025-10-11 13:25:16Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.controller]: DELETE_IN_PROGRESS  state changed
2025-10-11 09:25:45.481552 | hotstack | 2025-10-11 13:25:17Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.master0]: DELETE_IN_PROGRESS  state changed
2025-10-11 09:25:45.481565 | hotstack | 2025-10-11 13:25:18Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.networker1]: DELETE_IN_PROGRESS  state changed
2025-10-11 09:25:45.481578 | hotstack | 2025-10-11 13:25:19Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.machine-net-router-interface]: DELETE_COMPLETE  state changed
2025-10-11 09:25:45.481592 | hotstack | 2025-10-11 13:25:19Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.networker0]: DELETE_IN_PROGRESS  state changed
2025-10-11 09:25:45.481605 | hotstack | 2025-10-11 13:25:20Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.master2]: DELETE_IN_PROGRESS  state changed
2025-10-11 09:25:45.481618 | hotstack | 2025-10-11 13:25:21Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.ctlplane-net-router-interface]: DELETE_IN_PROGRESS  state changed
2025-10-11 09:25:45.481631 | hotstack | 2025-10-11 13:25:22Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.networker2]: DELETE_IN_PROGRESS  state changed
2025-10-11 09:25:45.481644 | hotstack | 2025-10-11 13:25:23Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.controller]: DELETE_COMPLETE  state changed
2025-10-11 09:25:45.481657 | hotstack | 2025-10-11 13:25:23Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.controller-init]: DELETE_IN_PROGRESS  state changed
2025-10-11 09:25:45.481670 | hotstack | 2025-10-11 13:25:23Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.controller-init]: DELETE_COMPLETE  state changed
2025-10-11 09:25:45.481684 | hotstack | 2025-10-11 13:25:23Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.controller-runcmd]: DELETE_IN_PROGRESS  state changed
2025-10-11 09:25:45.481697 | hotstack | 2025-10-11 13:25:24Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.controller-write-files]: DELETE_IN_PROGRESS  state changed
2025-10-11 09:25:45.481710 | hotstack | 2025-10-11 13:25:24Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.controller-write-files]: DELETE_COMPLETE  state changed
2025-10-11 09:25:45.481723 | hotstack | 2025-10-11 13:25:24Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.controller_users]: DELETE_IN_PROGRESS  state changed
2025-10-11 09:25:45.481736 | hotstack | 2025-10-11 13:25:24Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.master1]: DELETE_IN_PROGRESS  state changed
2025-10-11 09:25:45.481749 | hotstack | 2025-10-11 13:25:24Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.controller-runcmd]: DELETE_COMPLETE  state changed
2025-10-11 09:25:45.481763 | hotstack | 2025-10-11 13:25:24Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.controller_users]: DELETE_COMPLETE  state changed
2025-10-11 09:25:45.481791 | hotstack | 2025-10-11 13:25:24Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.ctlplane-net-router-interface]: DELETE_COMPLETE  state changed
2025-10-11 09:25:45.481805 | hotstack | 2025-10-11 13:25:24Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.compute0]: DELETE_IN_PROGRESS  state changed
2025-10-11 09:25:45.481819 | hotstack | 2025-10-11 13:25:26Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.ironic-net-router-interface]: DELETE_IN_PROGRESS  state changed
2025-10-11 09:25:45.481832 | hotstack | 2025-10-11 13:25:27Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.compute1]: DELETE_IN_PROGRESS  state changed
2025-10-11 09:25:45.481845 | hotstack | 2025-10-11 13:25:27Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.networker1]: DELETE_COMPLETE  state changed
2025-10-11 09:25:45.481858 | hotstack | 2025-10-11 13:25:28Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.networker1-trunk0]: DELETE_IN_PROGRESS  state changed
2025-10-11 09:25:45.481871 | hotstack | 2025-10-11 13:25:29Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.networker1-trunk0]: DELETE_COMPLETE  state changed
2025-10-11 09:25:45.481884 | hotstack | 2025-10-11 13:25:29Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.networker1-tenant-port]: DELETE_IN_PROGRESS  state changed
2025-10-11 09:25:45.481912 | hotstack | 2025-10-11 13:25:29Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.networker1-internal-api-port]: DELETE_IN_PROGRESS  state changed
2025-10-11 09:25:45.481926 | hotstack | 2025-10-11 13:25:29Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.networker1-ctlplane-trunk-parent-port]: DELETE_IN_PROGRESS  state changed
2025-10-11 09:25:45.481942 | hotstack | 2025-10-11 13:25:30Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.networker1-ctlplane-trunk-parent-port]: DELETE_COMPLETE  state changed
2025-10-11 09:25:45.481956 | hotstack | 2025-10-11 13:25:30Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.ironic-net-router-interface]: DELETE_COMPLETE  state changed
2025-10-11 09:25:45.481989 | hotstack | 2025-10-11 13:25:30Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.router]: DELETE_IN_PROGRESS  state changed
2025-10-11 09:25:45.482010 | hotstack | 2025-10-11 13:25:30Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.networker1-tenant-port]: DELETE_COMPLETE  state changed
2025-10-11 09:25:45.482041 | hotstack | 2025-10-11 13:25:31Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.networker1-internal-api-port]: DELETE_COMPLETE  state changed
2025-10-11 09:25:50.623223 | hotstack | 2025-10-11 13:25:31Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.networker0]: DELETE_COMPLETE  state changed
2025-10-11 09:25:50.623325 | hotstack | 2025-10-11 13:25:31Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.networker0-trunk0]: DELETE_IN_PROGRESS  state changed
2025-10-11 09:25:50.623351 | hotstack | 2025-10-11 13:25:32Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.networker2]: DELETE_COMPLETE  state changed
2025-10-11 09:25:50.623366 | hotstack | 2025-10-11 13:25:33Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.networker2-trunk0]: DELETE_IN_PROGRESS  state changed
2025-10-11 09:25:50.623380 | hotstack | 2025-10-11 13:25:33Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.networker_init]: DELETE_IN_PROGRESS  state changed
2025-10-11 09:25:50.623408 | hotstack | 2025-10-11 13:25:33Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.networker_init]: DELETE_COMPLETE  state changed
2025-10-11 09:25:50.623422 | hotstack | 2025-10-11 13:25:33Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.networker0-trunk0]: DELETE_COMPLETE  state changed
2025-10-11 09:25:50.623435 | hotstack | 2025-10-11 13:25:33Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.networker0-ctlplane-trunk-parent-port]: DELETE_IN_PROGRESS  state changed
2025-10-11 09:25:50.623450 | hotstack | 2025-10-11 13:25:33Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.networker0-internal-api-port]: DELETE_IN_PROGRESS  state changed
2025-10-11 09:25:50.623463 | hotstack | 2025-10-11 13:25:33Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.networker0-tenant-port]: DELETE_IN_PROGRESS  state changed
2025-10-11 09:25:50.623476 | hotstack | 2025-10-11 13:25:34Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.router]: DELETE_COMPLETE  state changed
2025-10-11 09:25:50.623490 | hotstack | 2025-10-11 13:25:34Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.networker0-tenant-port]: DELETE_COMPLETE  state changed
2025-10-11 09:25:50.623503 | hotstack | 2025-10-11 13:25:35Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.networker0-internal-api-port]: DELETE_COMPLETE  state changed
2025-10-11 09:25:50.623516 | hotstack | 2025-10-11 13:25:35Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.networker2-trunk0]: DELETE_COMPLETE  state changed
2025-10-11 09:25:50.623533 | hotstack | 2025-10-11 13:25:35Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.networker2-internal-api-port]: DELETE_IN_PROGRESS  state changed
2025-10-11 09:25:50.623547 | hotstack | 2025-10-11 13:25:35Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.networker2-tenant-port]: DELETE_IN_PROGRESS  state changed
2025-10-11 09:25:50.623560 | hotstack | 2025-10-11 13:25:35Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.networker2-ctlplane-trunk-parent-port]: DELETE_IN_PROGRESS  state changed
2025-10-11 09:25:50.623574 | hotstack | 2025-10-11 13:25:35Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.networker0-ctlplane-trunk-parent-port]: DELETE_COMPLETE  state changed
2025-10-11 09:25:50.623587 | hotstack | 2025-10-11 13:25:36Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.networker2-internal-api-port]: DELETE_COMPLETE  state changed
2025-10-11 09:25:50.623600 | hotstack | 2025-10-11 13:25:36Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.networker2-ctlplane-trunk-parent-port]: DELETE_COMPLETE  state changed
2025-10-11 09:25:50.623614 | hotstack | 2025-10-11 13:25:37Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.networker2-tenant-port]: DELETE_COMPLETE  state changed
2025-10-11 09:25:50.623642 | hotstack | 2025-10-11 13:25:38Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.compute0]: DELETE_COMPLETE  state changed
2025-10-11 09:25:50.623656 | hotstack | 2025-10-11 13:25:38Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.compute0_init]: DELETE_IN_PROGRESS  state changed
2025-10-11 09:25:50.623669 | hotstack | 2025-10-11 13:25:38Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.compute0_init]: DELETE_COMPLETE  state changed
2025-10-11 09:25:50.623682 | hotstack | 2025-10-11 13:25:38Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.compute0-trunk0]: DELETE_IN_PROGRESS  state changed
2025-10-11 09:25:50.623696 | hotstack | 2025-10-11 13:25:38Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.compute0_hostname]: DELETE_IN_PROGRESS  state changed
2025-10-11 09:25:50.623709 | hotstack | 2025-10-11 13:25:38Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.compute0_hostname]: DELETE_COMPLETE  state changed
2025-10-11 09:25:50.623731 | hotstack | 2025-10-11 13:25:39Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.compute0-trunk0]: DELETE_COMPLETE  state changed
2025-10-11 09:25:50.623744 | hotstack | 2025-10-11 13:25:39Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.compute0-storage-port]: DELETE_IN_PROGRESS  state changed
2025-10-11 09:25:50.623758 | hotstack | 2025-10-11 13:25:39Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.compute0-internal-api-port]: DELETE_IN_PROGRESS  state changed
2025-10-11 09:25:50.623771 | hotstack | 2025-10-11 13:25:39Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.compute0-tenant-port]: DELETE_IN_PROGRESS  state changed
2025-10-11 09:25:50.623784 | hotstack | 2025-10-11 13:25:40Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.compute0-ctlplane-trunk-parent-port]: DELETE_IN_PROGRESS  state changed
2025-10-11 09:25:50.623798 | hotstack | 2025-10-11 13:25:40Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.compute0-storage-port]: DELETE_COMPLETE  state changed
2025-10-11 09:25:50.623811 | hotstack | 2025-10-11 13:25:40Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.compute0-internal-api-port]: DELETE_COMPLETE  state changed
2025-10-11 09:25:50.623824 | hotstack | 2025-10-11 13:25:40Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.compute0-ctlplane-trunk-parent-port]: DELETE_COMPLETE  state changed
2025-10-11 09:25:50.623854 | hotstack | 2025-10-11 13:25:41Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.master2]: DELETE_COMPLETE  state changed
2025-10-11 09:25:50.623868 | hotstack | 2025-10-11 13:25:41Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.compute0-tenant-port]: DELETE_COMPLETE  state changed
2025-10-11 09:25:50.623882 | hotstack | 2025-10-11 13:25:41Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.master2-cinder-vol1]: DELETE_IN_PROGRESS  state changed
2025-10-11 09:25:50.623895 | hotstack | 2025-10-11 13:25:41Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.master2-lvms-vol0]: DELETE_IN_PROGRESS  state changed
2025-10-11 09:25:50.623911 | hotstack | 2025-10-11 13:25:41Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.master2-ironic-port]: DELETE_IN_PROGRESS  state changed
2025-10-11 09:25:50.623925 | hotstack | 2025-10-11 13:25:41Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.master2-cinder-vol2]: DELETE_IN_PROGRESS  state changed
2025-10-11 09:25:50.623939 | hotstack | 2025-10-11 13:25:41Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.master2-trunk0]: DELETE_IN_PROGRESS  state changed
2025-10-11 09:25:50.623952 | hotstack | 2025-10-11 13:25:41Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.master2-machine-port]: DELETE_IN_PROGRESS  state changed
2025-10-11 09:25:50.623965 | hotstack | 2025-10-11 13:25:41Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.master2-cinder-vol0]: DELETE_IN_PROGRESS  state changed
2025-10-11 09:25:50.623979 | hotstack | 2025-10-11 13:25:41Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.compute1]: DELETE_COMPLETE  state changed
2025-10-11 09:25:50.623992 | hotstack | 2025-10-11 13:25:41Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.compute1-trunk0]: DELETE_IN_PROGRESS  state changed
2025-10-11 09:25:50.624005 | hotstack | 2025-10-11 13:25:41Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.compute1_init]: DELETE_IN_PROGRESS  state changed
2025-10-11 09:25:50.624019 | hotstack | 2025-10-11 13:25:41Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.compute1_init]: DELETE_COMPLETE  state changed
2025-10-11 09:25:50.624032 | hotstack | 2025-10-11 13:25:41Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.compute1_hostname]: DELETE_IN_PROGRESS  state changed
2025-10-11 09:25:50.624045 | hotstack | 2025-10-11 13:25:41Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.dataplane_users]: DELETE_IN_PROGRESS  state changed
2025-10-11 09:25:50.624116 | hotstack | 2025-10-11 13:25:42Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.master0]: DELETE_COMPLETE  state changed
2025-10-11 09:25:50.624138 | hotstack | 2025-10-11 13:25:42Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.compute1_hostname]: DELETE_COMPLETE  state changed
2025-10-11 09:25:50.624151 | hotstack | 2025-10-11 13:25:42Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.dataplane_users]: DELETE_COMPLETE  state changed
2025-10-11 09:25:50.624165 | hotstack | 2025-10-11 13:25:42Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.master0-ironic-port]: DELETE_IN_PROGRESS  state changed
2025-10-11 09:25:50.624178 | hotstack | 2025-10-11 13:25:42Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.master0-cinder-vol0]: DELETE_IN_PROGRESS  state changed
2025-10-11 09:25:50.624192 | hotstack | 2025-10-11 13:25:42Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.master2-ironic-port]: DELETE_COMPLETE  state changed
2025-10-11 09:25:50.624209 | hotstack | 2025-10-11 13:25:42Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.master0-cinder-vol2]: DELETE_IN_PROGRESS  state changed
2025-10-11 09:25:50.624225 | hotstack | 2025-10-11 13:25:42Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.master0-lvms-vol0]: DELETE_IN_PROGRESS  state changed
2025-10-11 09:25:50.624239 | hotstack | 2025-10-11 13:25:42Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.master0-cinder-vol1]: DELETE_IN_PROGRESS  state changed
2025-10-11 09:25:50.624252 | hotstack | 2025-10-11 13:25:42Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.master0-trunk0]: DELETE_IN_PROGRESS  state changed
2025-10-11 09:25:50.624266 | hotstack | 2025-10-11 13:25:42Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.master0-machine-port]: DELETE_IN_PROGRESS  state changed
2025-10-11 09:25:50.624279 | hotstack | 2025-10-11 13:25:42Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.master2-machine-port]: DELETE_COMPLETE  state changed
2025-10-11 09:25:50.624292 | hotstack | 2025-10-11 13:25:42Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.master2-trunk0]: DELETE_COMPLETE  state changed
2025-10-11 09:25:50.624305 | hotstack | 2025-10-11 13:25:42Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.master2-octavia-port]: DELETE_IN_PROGRESS  state changed
2025-10-11 09:25:50.624319 | hotstack | 2025-10-11 13:25:42Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.master2-internal-api-port]: DELETE_IN_PROGRESS  state changed
2025-10-11 09:25:50.624332 | hotstack | 2025-10-11 13:25:42Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.master2-ctlplane-trunk-parent-port]: DELETE_IN_PROGRESS  state changed
2025-10-11 09:25:50.624346 | hotstack | 2025-10-11 13:25:42Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.master2-tenant-port]: DELETE_IN_PROGRESS  state changed
2025-10-11 09:25:50.624359 | hotstack | 2025-10-11 13:25:42Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.master2-storage-port]: DELETE_IN_PROGRESS  state changed
2025-10-11 09:25:50.624372 | hotstack | 2025-10-11 13:25:43Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.master0-ironic-port]: DELETE_COMPLETE  state changed
2025-10-11 09:25:50.624386 | hotstack | 2025-10-11 13:25:43Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.master0-cinder-vol0]: DELETE_COMPLETE  state changed
2025-10-11 09:25:50.624399 | hotstack | 2025-10-11 13:25:43Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.master0-cinder-vol1]: DELETE_COMPLETE  state changed
2025-10-11 09:25:50.624427 | hotstack | 2025-10-11 13:25:43Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.master0-cinder-vol2]: DELETE_COMPLETE  state changed
2025-10-11 09:26:00.820165 | hotstack | 2025-10-11 13:25:43Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.master0-lvms-vol0]: DELETE_COMPLETE  state changed
2025-10-11 09:26:00.820270 | hotstack | 2025-10-11 13:25:43Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.master0-machine-port]: DELETE_COMPLETE  state changed
2025-10-11 09:26:00.820310 | hotstack | 2025-10-11 13:25:44Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.compute1-trunk0]: DELETE_COMPLETE  state changed
2025-10-11 09:26:00.820325 | hotstack | 2025-10-11 13:25:44Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.master2-internal-api-port]: DELETE_COMPLETE  state changed
2025-10-11 09:26:00.820339 | hotstack | 2025-10-11 13:25:44Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.master2-storage-port]: DELETE_COMPLETE  state changed
2025-10-11 09:26:00.820382 | hotstack | 2025-10-11 13:25:44Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.master0-trunk0]: DELETE_COMPLETE  state changed
2025-10-11 09:26:00.820412 | hotstack | 2025-10-11 13:25:44Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.master2-ctlplane-trunk-parent-port]: DELETE_COMPLETE  state changed
2025-10-11 09:26:00.820440 | hotstack | 2025-10-11 13:25:44Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.compute1-internal-api-port]: DELETE_IN_PROGRESS  state changed
2025-10-11 09:26:00.820454 | hotstack | 2025-10-11 13:25:44Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.compute1-tenant-port]: DELETE_IN_PROGRESS  state changed
2025-10-11 09:26:00.820468 | hotstack | 2025-10-11 13:25:44Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.master2-tenant-port]: DELETE_COMPLETE  state changed
2025-10-11 09:26:00.820481 | hotstack | 2025-10-11 13:25:44Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.compute1-ctlplane-trunk-parent-port]: DELETE_IN_PROGRESS  state changed
2025-10-11 09:26:00.820495 | hotstack | 2025-10-11 13:25:44Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.master0-ctlplane-trunk-parent-port]: DELETE_IN_PROGRESS  state changed
2025-10-11 09:26:00.820514 | hotstack | 2025-10-11 13:25:44Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.compute1-storage-port]: DELETE_IN_PROGRESS  state changed
2025-10-11 09:26:00.820528 | hotstack | 2025-10-11 13:25:44Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.master2-octavia-port]: DELETE_COMPLETE  state changed
2025-10-11 09:26:00.820541 | hotstack | 2025-10-11 13:25:44Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.master0-tenant-port]: DELETE_IN_PROGRESS  state changed
2025-10-11 09:26:00.820555 | hotstack | 2025-10-11 13:25:44Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.master0-storage-port]: DELETE_IN_PROGRESS  state changed
2025-10-11 09:26:00.820568 | hotstack | 2025-10-11 13:25:44Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.master0-internal-api-port]: DELETE_IN_PROGRESS  state changed
2025-10-11 09:26:00.820581 | hotstack | 2025-10-11 13:25:44Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.master0-octavia-port]: DELETE_IN_PROGRESS  state changed
2025-10-11 09:26:00.820595 | hotstack | 2025-10-11 13:25:45Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.compute1-tenant-port]: DELETE_COMPLETE  state changed
2025-10-11 09:26:00.820609 | hotstack | 2025-10-11 13:25:45Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.compute1-storage-port]: DELETE_COMPLETE  state changed
2025-10-11 09:26:00.820623 | hotstack | 2025-10-11 13:25:45Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.master0-ctlplane-trunk-parent-port]: DELETE_COMPLETE  state changed
2025-10-11 09:26:00.820636 | hotstack | 2025-10-11 13:25:45Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.compute1-internal-api-port]: DELETE_COMPLETE  state changed
2025-10-11 09:26:00.820650 | hotstack | 2025-10-11 13:25:45Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.master0-storage-port]: DELETE_COMPLETE  state changed
2025-10-11 09:26:00.820663 | hotstack | 2025-10-11 13:25:45Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.master0-tenant-port]: DELETE_COMPLETE  state changed
2025-10-11 09:26:00.820676 | hotstack | 2025-10-11 13:25:45Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.master0-octavia-port]: DELETE_COMPLETE  state changed
2025-10-11 09:26:00.820707 | hotstack | 2025-10-11 13:25:45Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.master0-internal-api-port]: DELETE_COMPLETE  state changed
2025-10-11 09:26:00.820722 | hotstack | 2025-10-11 13:25:46Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.compute1-ctlplane-trunk-parent-port]: DELETE_COMPLETE  state changed
2025-10-11 09:26:00.820735 | hotstack | 2025-10-11 13:25:48Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.master1]: DELETE_COMPLETE  state changed
2025-10-11 09:26:00.820749 | hotstack | 2025-10-11 13:25:48Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.master1-ironic-port]: DELETE_IN_PROGRESS  state changed
2025-10-11 09:26:00.820762 | hotstack | 2025-10-11 13:25:48Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.master1-lvms-vol0]: DELETE_IN_PROGRESS  state changed
2025-10-11 09:26:00.820775 | hotstack | 2025-10-11 13:25:48Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.master1-cinder-vol2]: DELETE_IN_PROGRESS  state changed
2025-10-11 09:26:00.820789 | hotstack | 2025-10-11 13:25:48Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.master1-trunk0]: DELETE_IN_PROGRESS  state changed
2025-10-11 09:26:00.820802 | hotstack | 2025-10-11 13:25:48Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.master1-cinder-vol0]: DELETE_IN_PROGRESS  state changed
2025-10-11 09:26:00.820815 | hotstack | 2025-10-11 13:25:48Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.master1-cinder-vol1]: DELETE_IN_PROGRESS  state changed
2025-10-11 09:26:00.820829 | hotstack | 2025-10-11 13:25:48Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.master1-machine-port]: DELETE_IN_PROGRESS  state changed
2025-10-11 09:26:00.820851 | hotstack | 2025-10-11 13:25:48Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.master1-ironic-port]: DELETE_COMPLETE  state changed
2025-10-11 09:26:00.820864 | hotstack | 2025-10-11 13:25:48Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.ironic-subnet]: DELETE_IN_PROGRESS  state changed
2025-10-11 09:26:00.820878 | hotstack | 2025-10-11 13:25:49Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.master1-lvms-vol0]: DELETE_COMPLETE  state changed
2025-10-11 09:26:00.820891 | hotstack | 2025-10-11 13:25:49Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.ironic-subnet]: DELETE_COMPLETE  state changed
2025-10-11 09:26:00.820905 | hotstack | 2025-10-11 13:25:49Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.master1-cinder-vol2]: DELETE_COMPLETE  state changed
2025-10-11 09:26:00.820918 | hotstack | 2025-10-11 13:25:49Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.master1-machine-port]: DELETE_COMPLETE  state changed
2025-10-11 09:26:00.820931 | hotstack | 2025-10-11 13:25:49Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.master1-cinder-vol0]: DELETE_COMPLETE  state changed
2025-10-11 09:26:00.820945 | hotstack | 2025-10-11 13:25:49Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.ironic-net]: DELETE_IN_PROGRESS  state changed
2025-10-11 09:26:00.820961 | hotstack | 2025-10-11 13:25:49Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.master1-trunk0]: DELETE_COMPLETE  state changed
2025-10-11 09:26:00.820979 | hotstack | 2025-10-11 13:25:49Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.master1-cinder-vol1]: DELETE_COMPLETE  state changed
2025-10-11 09:26:00.820997 | hotstack | 2025-10-11 13:25:49Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.master1-internal-api-port]: DELETE_IN_PROGRESS  state changed
2025-10-11 09:26:00.821015 | hotstack | 2025-10-11 13:25:49Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.extra-dhcp-opts-value]: DELETE_IN_PROGRESS  state changed
2025-10-11 09:26:00.821032 | hotstack | 2025-10-11 13:25:49Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.master1-storage-port]: DELETE_IN_PROGRESS  state changed
2025-10-11 09:26:00.821045 | hotstack | 2025-10-11 13:25:49Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.extra-dhcp-opts-value]: DELETE_COMPLETE  state changed
2025-10-11 09:26:00.821122 | hotstack | 2025-10-11 13:25:49Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.master1-octavia-port]: DELETE_IN_PROGRESS  state changed
2025-10-11 09:26:00.821148 | hotstack | 2025-10-11 13:25:49Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.controller-machine-port]: DELETE_IN_PROGRESS  state changed
2025-10-11 09:26:00.821162 | hotstack | 2025-10-11 13:25:49Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.master1-tenant-port]: DELETE_IN_PROGRESS  state changed
2025-10-11 09:26:00.821176 | hotstack | 2025-10-11 13:25:49Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.master1-ctlplane-trunk-parent-port]: DELETE_IN_PROGRESS  state changed
2025-10-11 09:26:00.821189 | hotstack | 2025-10-11 13:25:50Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.master1-storage-port]: DELETE_COMPLETE  state changed
2025-10-11 09:26:00.821203 | hotstack | 2025-10-11 13:25:50Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.storage-subnet]: DELETE_IN_PROGRESS  state changed
2025-10-11 09:26:00.821216 | hotstack | 2025-10-11 13:25:50Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.master1-internal-api-port]: DELETE_COMPLETE  state changed
2025-10-11 09:26:00.821229 | hotstack | 2025-10-11 13:25:50Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.master1-tenant-port]: DELETE_COMPLETE  state changed
2025-10-11 09:26:00.821243 | hotstack | 2025-10-11 13:25:50Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.master1-octavia-port]: DELETE_COMPLETE  state changed
2025-10-11 09:26:00.821256 | hotstack | 2025-10-11 13:25:50Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.ironic-net]: DELETE_COMPLETE  state changed
2025-10-11 09:26:00.821269 | hotstack | 2025-10-11 13:25:50Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.tenant-subnet]: DELETE_IN_PROGRESS  state changed
2025-10-11 09:26:00.821282 | hotstack | 2025-10-11 13:25:50Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.internal-api-subnet]: DELETE_IN_PROGRESS  state changed
2025-10-11 09:26:00.821308 | hotstack | 2025-10-11 13:25:50Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.controller-machine-port]: DELETE_COMPLETE  state changed
2025-10-11 09:26:00.899382 | hotstack | 2025-10-11 13:25:50Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.octavia-subnet]: DELETE_IN_PROGRESS  state changed
2025-10-11 09:26:00.899672 | hotstack | 2025-10-11 13:25:50Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.machine-subnet]: DELETE_IN_PROGRESS  state changed
2025-10-11 09:26:00.899695 | hotstack | 2025-10-11 13:25:51Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.storage-subnet]: DELETE_COMPLETE  state changed
2025-10-11 09:26:00.899727 | hotstack | 2025-10-11 13:25:51Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.master1-ctlplane-trunk-parent-port]: DELETE_COMPLETE  state changed
2025-10-11 09:26:00.899742 | hotstack | 2025-10-11 13:25:51Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.storage-net]: DELETE_IN_PROGRESS  state changed
2025-10-11 09:26:00.899755 | hotstack | 2025-10-11 13:25:51Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.ctlplane-subnet]: DELETE_IN_PROGRESS  state changed
2025-10-11 09:26:00.899769 | hotstack | 2025-10-11 13:25:51Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.tenant-subnet]: DELETE_COMPLETE  state changed
2025-10-11 09:26:00.899782 | hotstack | 2025-10-11 13:25:51Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.internal-api-subnet]: DELETE_COMPLETE  state changed
2025-10-11 09:26:00.899795 | hotstack | 2025-10-11 13:25:51Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.octavia-subnet]: DELETE_COMPLETE  state changed
2025-10-11 09:26:00.899809 | hotstack | 2025-10-11 13:25:51Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.tenant-net]: DELETE_IN_PROGRESS  state changed
2025-10-11 09:26:00.899834 | hotstack | 2025-10-11 13:25:51Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.internal-api-net]: DELETE_IN_PROGRESS  state changed
2025-10-11 09:26:00.899847 | hotstack | 2025-10-11 13:25:51Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.octavia-net]: DELETE_IN_PROGRESS  state changed
2025-10-11 09:26:00.899861 | hotstack | 2025-10-11 13:25:51Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.machine-subnet]: DELETE_COMPLETE  state changed
2025-10-11 09:26:00.899874 | hotstack | 2025-10-11 13:25:51Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.machine-net]: DELETE_IN_PROGRESS  state changed
2025-10-11 09:26:00.899888 | hotstack | 2025-10-11 13:25:51Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.ctlplane-subnet]: DELETE_COMPLETE  state changed
2025-10-11 09:26:00.899901 | hotstack | 2025-10-11 13:25:51Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.ctlplane-net]: DELETE_IN_PROGRESS  state changed
2025-10-11 09:26:00.899914 | hotstack | 2025-10-11 13:25:52Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.storage-net]: DELETE_COMPLETE  state changed
2025-10-11 09:26:00.899927 | hotstack | 2025-10-11 13:25:52Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.machine-net]: DELETE_COMPLETE  state changed
2025-10-11 09:26:00.899940 | hotstack | 2025-10-11 13:25:52Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.internal-api-net]: DELETE_COMPLETE  state changed
2025-10-11 09:26:00.899954 | hotstack | 2025-10-11 13:25:52Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.tenant-net]: DELETE_COMPLETE  state changed
2025-10-11 09:26:00.899967 | hotstack | 2025-10-11 13:25:53Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.ctlplane-net]: DELETE_COMPLETE  state changed
2025-10-11 09:26:00.899980 | hotstack | 2025-10-11 13:25:53Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.octavia-net]: DELETE_COMPLETE  state changed
2025-10-11 09:26:00.899993 | hotstack | 2025-10-11 13:25:58Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.master2-cinder-vol1]: DELETE_COMPLETE  state changed
2025-10-11 09:26:00.900006 | hotstack | 2025-10-11 13:25:58Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.master2-lvms-vol0]: DELETE_COMPLETE  state changed
2025-10-11 09:26:00.900020 | hotstack | 2025-10-11 13:25:58Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.master2-cinder-vol2]: DELETE_COMPLETE  state changed
2025-10-11 09:26:00.900033 | hotstack | 2025-10-11 13:25:58Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b.master2-cinder-vol0]: DELETE_COMPLETE  state changed
2025-10-11 09:26:00.900046 | hotstack | 2025-10-11 13:25:58Z [hs-2eeebd5c21844fdd9eefbaf9d59e8d1b]: DELETE_COMPLETE  Stack DELETE completed successfully
2025-10-11 13:26:01.405491 | hotstack | changed
2025-10-11 13:26:01.416601 | 
2025-10-11 13:26:01.416757 | TASK [Print message that stack failed to delete]
2025-10-11 13:26:01.444087 | hotstack | skipping: Conditional result was False
2025-10-11 13:26:01.454449 | 
2025-10-11 13:26:01.454648 | TASK [Print message that stack deleted successfully]
2025-10-11 13:26:01.501926 | hotstack | ok: Stack deleted successfully.
2025-10-11 13:26:01.510017 | 
2025-10-11 13:26:01.510146 | TASK [Delete clouds.yaml]
2025-10-11 13:26:01.789042 | hotstack | Output suppressed because no_log was given
2025-10-11 13:26:01.807865 | 
2025-10-11 13:26:01.807917 | PLAY RECAP
2025-10-11 13:26:01.807960 | hotstack | ok: 18 changed: 8 unreachable: 0 failed: 0 skipped: 3 rescued: 2 ignored: 1
2025-10-11 13:26:01.807981 | 
2025-10-11 13:26:01.901197 | POST-RUN END RESULT_NORMAL: [trusted : review.rdoproject.org/config/playbooks/hotstack/post.yaml@master]
2025-10-11 13:26:01.903893 | POST-RUN START: [trusted : review.rdoproject.org/config/playbooks/base-minimal/post-ssh.yaml@master]
2025-10-11 13:26:02.484689 | 
2025-10-11 13:26:02.484858 | PLAY [all]
2025-10-11 13:26:02.503455 | 
2025-10-11 13:26:02.503569 | TASK [include_role : fetch-output]
2025-10-11 13:26:02.532368 | hotstack | ok
2025-10-11 13:26:02.548793 | 
2025-10-11 13:26:02.548903 | TASK [fetch-output : Set log path for multiple nodes]
2025-10-11 13:26:02.604240 | hotstack | skipping: Conditional result was False
2025-10-11 13:26:02.609744 | 
2025-10-11 13:26:02.609831 | TASK [fetch-output : Set log path for single node]
2025-10-11 13:26:02.648702 | hotstack | ok
2025-10-11 13:26:02.653830 | 
2025-10-11 13:26:02.653899 | LOOP [fetch-output : Ensure local output dirs]
2025-10-11 13:26:03.065844 | hotstack -> localhost | ok: "/var/lib/zuul/builds/2eeebd5c21844fdd9eefbaf9d59e8d1b/work/logs"
2025-10-11 13:26:03.262296 | hotstack -> localhost | changed: "/var/lib/zuul/builds/2eeebd5c21844fdd9eefbaf9d59e8d1b/work/artifacts"
2025-10-11 13:26:03.492693 | hotstack -> localhost | changed: "/var/lib/zuul/builds/2eeebd5c21844fdd9eefbaf9d59e8d1b/work/docs"
2025-10-11 13:26:03.507176 | 
2025-10-11 13:26:03.507256 | LOOP [fetch-output : Collect logs, artifacts and docs]
2025-10-11 13:26:04.169080 | hotstack | changed:
2025-10-11 13:26:04.169396 | hotstack | .d..t...... ./
2025-10-11 13:26:04.169465 | hotstack | cd+++++++++ hotlogs/
2025-10-11 13:26:04.169511 | hotstack | cd+++++++++ hotlogs/data/
2025-10-11 13:26:04.169555 | hotstack | cd+++++++++ hotlogs/manifests/
2025-10-11 13:26:04.169597 | hotstack | cd+++++++++ hotlogs/ocp_cluster/
2025-10-11 13:26:04.169638 | hotstack | >f+++++++++ hotlogs/ocp_cluster/openshift_install.log
2025-10-11 13:26:04.169687 | hotstack | cd+++++++++ hotlogs/ocp_cluster/butane/
2025-10-11 13:26:04.169745 | hotstack | >f+++++++++ hotlogs/ocp_cluster/butane/90-master-disable-netifnames.bu
2025-10-11 13:26:04.169842 | hotstack | >f+++++++++ hotlogs/ocp_cluster/butane/90-master-enable-iscsi.bu
2025-10-11 13:26:04.169884 | hotstack | >f+++++++++ hotlogs/ocp_cluster/butane/91-master-enable-multipath.bu
2025-10-11 13:26:04.169923 | hotstack | >f+++++++++ hotlogs/ocp_cluster/butane/92-master-lv-cinder-volumes.bu
2025-10-11 13:26:04.169960 | hotstack | cd+++++++++ hotlogs/ocp_cluster/config-assets/
2025-10-11 13:26:04.169998 | hotstack | >f+++++++++ hotlogs/ocp_cluster/config-assets/95-etcd_config.yaml
2025-10-11 13:26:04.170035 | hotstack | >f+++++++++ hotlogs/ocp_cluster/config-assets/ovn_k8s_config.yaml
2025-10-11 13:26:04.170073 | hotstack | cd+++++++++ hotlogs/ocp_cluster/machine-configs/
2025-10-11 13:26:04.170116 | hotstack | >f+++++++++ hotlogs/ocp_cluster/machine-configs/90-master-disable-netifnames.yaml
2025-10-11 13:26:04.170156 | hotstack | >f+++++++++ hotlogs/ocp_cluster/machine-configs/90-master-enable-iscsi.yaml
2025-10-11 13:26:04.170195 | hotstack | >f+++++++++ hotlogs/ocp_cluster/machine-configs/91-master-enable-multipath.yaml
2025-10-11 13:26:04.170232 | hotstack | >f+++++++++ hotlogs/ocp_cluster/machine-configs/92-master-lv-cinder-volumes.yaml
2025-10-11 13:26:04.170302 | hotstack | changed: All items complete
2025-10-11 13:26:04.170349 | 
2025-10-11 13:26:04.741201 | hotstack | changed: .d..t...... ./
2025-10-11 13:26:05.260868 | hotstack | changed: .d..t...... ./
2025-10-11 13:26:05.280715 | 
2025-10-11 13:26:05.280878 | TASK [include_role : fetch-output-openshift]
2025-10-11 13:26:05.305174 | hotstack | skipping: Conditional result was False
2025-10-11 13:26:05.311424 | 
2025-10-11 13:26:05.311491 | LOOP [merge-output-to-logs : Move artifacts and docs to logs dir]
2025-10-11 13:26:05.802445 | hotstack -> localhost | ok: Item: artifacts Runtime: 0:00:00.013755
2025-10-11 13:26:06.084946 | hotstack -> localhost | ok: Item: docs Runtime: 0:00:00.012867
2025-10-11 13:26:06.128458 | 
2025-10-11 13:26:06.128593 | PLAY [all]
2025-10-11 13:26:06.144312 | 
2025-10-11 13:26:06.144448 | TASK [remove-build-sshkey : Remove the build SSH key from all nodes]
2025-10-11 13:26:06.827364 | hotstack | changed
2025-10-11 13:26:06.866461 | 
2025-10-11 13:26:06.866682 | PLAY RECAP
2025-10-11 13:26:06.866800 | hotstack | ok: 5 changed: 4 unreachable: 0 failed: 0 skipped: 2 rescued: 0 ignored: 0
2025-10-11 13:26:06.866861 | 
2025-10-11 13:26:07.081547 | POST-RUN END RESULT_NORMAL: [trusted : review.rdoproject.org/config/playbooks/base-minimal/post-ssh.yaml@master]
2025-10-11 13:26:07.084671 | POST-RUN START: [trusted : review.rdoproject.org/config/playbooks/base-minimal/post-logs.yaml@master]
2025-10-11 13:26:07.695043 | 
2025-10-11 13:26:07.695166 | PLAY [localhost]
2025-10-11 13:26:07.713040 | 
2025-10-11 13:26:07.713153 | TASK [generate-zuul-manifest : Generate Zuul manifest]
2025-10-11 13:26:08.055989 | localhost | changed
2025-10-11 13:26:08.060537 | 
2025-10-11 13:26:08.060628 | TASK [generate-zuul-manifest : Return Zuul manifest URL to Zuul]
2025-10-11 13:26:08.084206 | localhost | ok
2025-10-11 13:26:08.091983 | 
2025-10-11 13:26:08.092047 | TASK [add-fileserver : Create SSH private key tempfile]
2025-10-11 13:26:08.454404 | localhost | changed
2025-10-11 13:26:08.459162 | 
2025-10-11 13:26:08.459228 | TASK [add-fileserver : Create SSH private key from secret]
2025-10-11 13:26:09.201220 | localhost | changed
2025-10-11 13:26:09.212154 | 
2025-10-11 13:26:09.212288 | TASK [add-fileserver : Add fileserver ssh key]
2025-10-11 13:26:09.652295 | localhost | Identity added: /var/lib/zuul/builds/2eeebd5c21844fdd9eefbaf9d59e8d1b/work/tmp/ansible.s5e57rrf (/var/lib/zuul/builds/2eeebd5c21844fdd9eefbaf9d59e8d1b/work/tmp/ansible.s5e57rrf)
2025-10-11 13:26:09.652490 | localhost | ok: Runtime: 0:00:00.012527
2025-10-11 13:26:09.656771 | 
2025-10-11 13:26:09.656835 | TASK [add-fileserver : Remove SSH private key from disk]
2025-10-11 13:26:09.973141 | localhost | ok: Runtime: 0:00:00.005949
2025-10-11 13:26:09.983274 | 
2025-10-11 13:26:09.983410 | TASK [add-fileserver : Add fileserver to inventory]
2025-10-11 13:26:10.050801 | localhost | changed
2025-10-11 13:26:10.060028 | 
2025-10-11 13:26:10.060169 | TASK [add-fileserver : Add fileserver server to known hosts]
2025-10-11 13:26:10.456662 | localhost | changed
2025-10-11 13:26:10.496381 | 
2025-10-11 13:26:10.496488 | PLAY [localhost]
2025-10-11 13:26:10.513307 | 
2025-10-11 13:26:10.513384 | TASK [Generate bulk log download script]
2025-10-11 13:26:10.558324 | localhost | ok
2025-10-11 13:26:10.571754 | 
2025-10-11 13:26:10.571819 | TASK [local-log-download : Check API endpoint is defined]
2025-10-11 13:26:10.610130 | localhost | ok: All assertions passed
2025-10-11 13:26:10.614385 | 
2025-10-11 13:26:10.614444 | TASK [local-log-download : Create download script]
2025-10-11 13:26:11.070857 | localhost -> localhost | changed
2025-10-11 13:26:11.089653 | 
2025-10-11 13:26:11.089820 | TASK [Register quick-download link]
2025-10-11 13:26:11.115037 | localhost | ok
2025-10-11 13:26:11.146851 | 
2025-10-11 13:26:11.146933 | PLAY [logserver.rdoproject.org]
2025-10-11 13:26:11.159240 | 
2025-10-11 13:26:11.159320 | TASK [Set zuul-log-path fact]
2025-10-11 13:26:11.177179 | logserver.rdoproject.org | ok
2025-10-11 13:26:11.189474 | 
2025-10-11 13:26:11.189555 | TASK [set-zuul-log-path-fact : Set log path for a build]
2025-10-11 13:26:11.228963 | logserver.rdoproject.org | ok
2025-10-11 13:26:11.240352 | 
2025-10-11 13:26:11.240479 | TASK [upload-logs : Create log directories]
2025-10-11 13:26:12.788793 | logserver.rdoproject.org | changed
2025-10-11 13:26:12.791817 | 
2025-10-11 13:26:12.791878 | TASK [upload-logs : Ensure logs are readable before uploading]
2025-10-11 13:26:13.071871 | logserver.rdoproject.org -> localhost | ok: Runtime: 0:00:00.008921
2025-10-11 13:26:13.081093 | 
2025-10-11 13:26:13.081223 | TASK [upload-logs : Upload logs to log server]
2025-10-11 13:26:13.820983 | logserver.rdoproject.org | Output suppressed because no_log was given
2025-10-11 13:26:13.826993 | 
2025-10-11 13:26:13.827144 | LOOP [upload-logs : Compress console log and json output]
2025-10-11 13:26:13.894051 | logserver.rdoproject.org | skipping: Conditional result was False
2025-10-11 13:26:13.906091 | logserver.rdoproject.org | skipping: Conditional result was False
2025-10-11 13:26:13.923612 | 
2025-10-11 13:26:13.923852 | LOOP [upload-logs : Upload compressed console log and json output]
2025-10-11 13:26:13.970717 | logserver.rdoproject.org | skipping: Conditional result was False
2025-10-11 13:26:13.971070 | 
2025-10-11 13:26:13.974361 | logserver.rdoproject.org | skipping: Conditional result was False
2025-10-11 13:26:13.989585 | 
2025-10-11 13:26:13.989808 | LOOP [upload-logs : Upload console log and json output]