2025-10-11 15:08:37.224504 | Job console starting...
2025-10-11 15:08:37.237787 | Updating repositories
2025-10-11 15:08:37.263286 | Preparing job workspace
2025-10-11 15:08:42.657520 | Running Ansible setup...
2025-10-11 15:08:46.647830 | PRE-RUN START: [trusted : review.rdoproject.org/config/playbooks/base-minimal/pre.yaml@master]
2025-10-11 15:08:47.223848 | 
2025-10-11 15:08:47.223956 | PLAY [localhost]
2025-10-11 15:08:47.232337 | 
2025-10-11 15:08:47.232403 | TASK [Gathering Facts]
2025-10-11 15:08:48.279239 | localhost | ok
2025-10-11 15:08:48.307499 | 
2025-10-11 15:08:48.307653 | TASK [log-inventory : Ensure Zuul Ansible directory exists]
2025-10-11 15:08:48.753155 | localhost -> localhost | changed
2025-10-11 15:08:48.763831 | 
2025-10-11 15:08:48.763973 | TASK [log-inventory : Copy ansible inventory to logs dir]
2025-10-11 15:08:49.688506 | localhost -> localhost | changed
2025-10-11 15:08:49.696711 | 
2025-10-11 15:08:49.696774 | TASK [Setup log path fact]
2025-10-11 15:08:49.719872 | localhost | ok
2025-10-11 15:08:49.734597 | 
2025-10-11 15:08:49.734701 | TASK [set-zuul-log-path-fact : Set log path for a build]
2025-10-11 15:08:49.764858 | localhost | ok
2025-10-11 15:08:49.774940 | 
2025-10-11 15:08:49.775026 | TASK [emit-job-header : Print job information]
2025-10-11 15:08:49.827635 | # Job Information
2025-10-11 15:08:49.827892 | Ansible Version: 2.15.12
2025-10-11 15:08:49.827949 | Job: vexxhost-hotstack-secret-rotate
2025-10-11 15:08:49.827992 | Pipeline: github-experimental-trigger
2025-10-11 15:08:49.828032 | Executor: ze04.softwarefactory-project.io
2025-10-11 15:08:49.828071 | Triggered by: https://github.com/openstack-k8s-operators/hotstack/pull/256
2025-10-11 15:08:49.828112 | Log URL (when completed): https://logserver.rdoproject.org/918/rdoproject.org/918b704357334bf18129d9b4a9d53906/
2025-10-11 15:08:49.828153 | Event ID: 035d36d0-a6b4-11f0-8ae7-5c53a359b9ce
2025-10-11 15:08:49.835862 | 
2025-10-11 15:08:49.835988 | LOOP [emit-job-header : Print node information]
2025-10-11 15:08:49.953567 | localhost | ok:
2025-10-11 15:08:49.954373 | localhost | # Node Information
2025-10-11 15:08:49.954450 | localhost | Inventory Hostname: hotstack
2025-10-11 15:08:49.954510 | localhost | Hostname: np0005481205
2025-10-11 15:08:49.954561 | localhost | Username: zuul
2025-10-11 15:08:49.954621 | localhost | Distro: CentOS 9
2025-10-11 15:08:49.954670 | localhost | Provider: vexxhost-nodepool-tripleo
2025-10-11 15:08:49.954777 | localhost | Region: RegionOne
2025-10-11 15:08:49.954824 | localhost | Label: cloud-centos-9-stream-tripleo-vexxhost
2025-10-11 15:08:49.954871 | localhost | Product Name: OpenStack Nova
2025-10-11 15:08:49.954917 | localhost | Interface IP: 38.102.83.70
2025-10-11 15:08:49.995014 | 
2025-10-11 15:08:49.995142 | PLAY [all]
2025-10-11 15:08:50.008446 | 
2025-10-11 15:08:50.008574 | TASK [Gather network facts]
2025-10-11 15:08:50.598340 | hotstack | ok
2025-10-11 15:08:50.629197 | 
2025-10-11 15:08:50.629326 | TASK [include_role : start-zuul-console]
2025-10-11 15:08:50.653205 | hotstack | ok
2025-10-11 15:08:50.668206 | 
2025-10-11 15:08:50.668288 | TASK [start-zuul-console : Start zuul_console daemon.]
2025-10-11 15:08:51.119331 | hotstack | ok
2025-10-11 15:08:51.137822 | 
2025-10-11 15:08:51.137945 | TASK [include_role : add-build-sshkey]
2025-10-11 15:08:51.174183 | hotstack | ok
2025-10-11 15:08:51.201097 | 
2025-10-11 15:08:51.201229 | TASK [add-build-sshkey : Check to see if ssh key was already created for this build]
2025-10-11 15:08:51.480703 | hotstack -> localhost | ok
2025-10-11 15:08:51.486430 | 
2025-10-11 15:08:51.486490 | TASK [add-build-sshkey : Create a new key in workspace based on build UUID]
2025-10-11 15:08:51.516558 | hotstack | ok
2025-10-11 15:08:51.528903 | hotstack | included: /var/lib/zuul/builds/918b704357334bf18129d9b4a9d53906/trusted/project_1/opendev.org/zuul/zuul-jobs/roles/add-build-sshkey/tasks/create-key-and-replace.yaml
2025-10-11 15:08:51.534354 | 
2025-10-11 15:08:51.534413 | TASK [add-build-sshkey : Create Temp SSH key]
2025-10-11 15:08:52.236010 | hotstack -> localhost | Generating public/private rsa key pair.
2025-10-11 15:08:52.236399 | hotstack -> localhost | Your identification has been saved in /var/lib/zuul/builds/918b704357334bf18129d9b4a9d53906/work/918b704357334bf18129d9b4a9d53906_id_rsa.
2025-10-11 15:08:52.236465 | hotstack -> localhost | Your public key has been saved in /var/lib/zuul/builds/918b704357334bf18129d9b4a9d53906/work/918b704357334bf18129d9b4a9d53906_id_rsa.pub.
2025-10-11 15:08:52.236513 | hotstack -> localhost | The key fingerprint is:
2025-10-11 15:08:52.236555 | hotstack -> localhost | SHA256:k/FXSmpO8bXIVT+ig67ynkh4uCvMj07i3dB0IR7Py3Y zuul-build-sshkey
2025-10-11 15:08:52.236596 | hotstack -> localhost | The key's randomart image is:
2025-10-11 15:08:52.236637 | hotstack -> localhost | +---[RSA 3072]----+
2025-10-11 15:08:52.236706 | hotstack -> localhost | |                .|
2025-10-11 15:08:52.236759 | hotstack -> localhost | |               ..|
2025-10-11 15:08:52.236800 | hotstack -> localhost | |     o .. . ..+..|
2025-10-11 15:08:52.236840 | hotstack -> localhost | |    . = .+.*.*...|
2025-10-11 15:08:52.236879 | hotstack -> localhost | |     o +S.=o* .  |
2025-10-11 15:08:52.236917 | hotstack -> localhost | |    = o o= ..    |
2025-10-11 15:08:52.236954 | hotstack -> localhost | |+. + + + E.      |
2025-10-11 15:08:52.236992 | hotstack -> localhost | |++o *.o +        |
2025-10-11 15:08:52.237029 | hotstack -> localhost | |.+o=.o+=         |
2025-10-11 15:08:52.237066 | hotstack -> localhost | +----[SHA256]-----+
2025-10-11 15:08:52.237185 | hotstack -> localhost | ok: Runtime: 0:00:00.230786
2025-10-11 15:08:52.249139 | 
2025-10-11 15:08:52.249275 | TASK [add-build-sshkey : Remote setup ssh keys (linux)]
2025-10-11 15:08:52.286263 | hotstack | ok
2025-10-11 15:08:52.306360 | hotstack | included: /var/lib/zuul/builds/918b704357334bf18129d9b4a9d53906/trusted/project_1/opendev.org/zuul/zuul-jobs/roles/add-build-sshkey/tasks/remote-linux.yaml
2025-10-11 15:08:52.321427 | 
2025-10-11 15:08:52.321510 | TASK [add-build-sshkey : Remove previously added zuul-build-sshkey]
2025-10-11 15:08:52.346608 | hotstack | skipping: Conditional result was False
2025-10-11 15:08:52.355253 | 
2025-10-11 15:08:52.355335 | TASK [add-build-sshkey : Enable access via build key on all nodes]
2025-10-11 15:08:52.827545 | hotstack | changed
2025-10-11 15:08:52.837906 | 
2025-10-11 15:08:52.838125 | TASK [add-build-sshkey : Make sure user has a .ssh]
2025-10-11 15:08:53.162481 | hotstack | ok
2025-10-11 15:08:53.172950 | 
2025-10-11 15:08:53.173074 | TASK [add-build-sshkey : Install build private key as SSH key on all nodes]
2025-10-11 15:08:54.177410 | hotstack | changed
2025-10-11 15:08:54.190975 | 
2025-10-11 15:08:54.191106 | TASK [add-build-sshkey : Install build public key as SSH key on all nodes]
2025-10-11 15:08:55.137825 | hotstack | changed
2025-10-11 15:08:55.148321 | 
2025-10-11 15:08:55.148450 | TASK [add-build-sshkey : Remote setup ssh keys (windows)]
2025-10-11 15:08:55.175600 | hotstack | skipping: Conditional result was False
2025-10-11 15:08:55.187391 | 
2025-10-11 15:08:55.187523 | TASK [remove-zuul-sshkey : Remove master key from local agent]
2025-10-11 15:08:55.612445 | hotstack -> localhost | changed
2025-10-11 15:08:55.633139 | 
2025-10-11 15:08:55.633269 | TASK [add-build-sshkey : Add back temp key]
2025-10-11 15:08:55.976232 | hotstack -> localhost | Identity added: /var/lib/zuul/builds/918b704357334bf18129d9b4a9d53906/work/918b704357334bf18129d9b4a9d53906_id_rsa (zuul-build-sshkey)
2025-10-11 15:08:55.976564 | hotstack -> localhost | ok: Runtime: 0:00:00.014961
2025-10-11 15:08:55.988284 | 
2025-10-11 15:08:55.988436 | TASK [add-build-sshkey : Verify we can still SSH to all nodes]
2025-10-11 15:08:56.403777 | hotstack | ok
2025-10-11 15:08:56.446926 | 
2025-10-11 15:08:56.447068 | TASK [add-build-sshkey : Verify we can still SSH to all nodes (windows)]
2025-10-11 15:08:56.482702 | hotstack | skipping: Conditional result was False
2025-10-11 15:08:56.495419 | 
2025-10-11 15:08:56.495501 | TASK [include_role : validate-host]
2025-10-11 15:08:56.529619 | hotstack | ok
2025-10-11 15:08:56.561502 | 
2025-10-11 15:08:56.561589 | TASK [validate-host : Define zuul_info_dir fact]
2025-10-11 15:08:56.606051 | hotstack | ok
2025-10-11 15:08:56.616151 | 
2025-10-11 15:08:56.616281 | TASK [validate-host : Ensure Zuul Ansible directory exists]
2025-10-11 15:08:56.877253 | hotstack -> localhost | ok
2025-10-11 15:08:56.889819 | 
2025-10-11 15:08:56.889952 | TASK [validate-host : Collect information about the host]
2025-10-11 15:08:57.734887 | hotstack | ok
2025-10-11 15:08:57.753490 | 
2025-10-11 15:08:57.753630 | TASK [validate-host : Sanitize hostname]
2025-10-11 15:08:57.830157 | hotstack | ok
2025-10-11 15:08:57.840597 | 
2025-10-11 15:08:57.840760 | TASK [validate-host : Write out all ansible variables/facts known for each host]
2025-10-11 15:08:58.360187 | hotstack -> localhost | changed
2025-10-11 15:08:58.375504 | 
2025-10-11 15:08:58.375659 | TASK [validate-host : Collect information about zuul worker]
2025-10-11 15:08:58.861583 | hotstack | ok
2025-10-11 15:08:58.871813 | 
2025-10-11 15:08:58.871950 | TASK [validate-host : Write out all zuul information for each host]
2025-10-11 15:08:59.375576 | hotstack -> localhost | changed
2025-10-11 15:08:59.396048 | 
2025-10-11 15:08:59.396189 | TASK [include_role : prepare-workspace-openshift]
2025-10-11 15:08:59.422996 | hotstack | skipping: Conditional result was False
2025-10-11 15:08:59.434183 | 
2025-10-11 15:08:59.434316 | TASK [include_role : remove-zuul-sshkey]
2025-10-11 15:08:59.460753 | hotstack | skipping: Conditional result was False
2025-10-11 15:08:59.470977 | 
2025-10-11 15:08:59.471105 | LOOP [ensure-output-dirs : Empty Zuul Output directories by removing them]
2025-10-11 15:08:59.781707 | hotstack | ok: "logs"
2025-10-11 15:08:59.781952 | hotstack | ok: All items complete
2025-10-11 15:08:59.781978 | 
2025-10-11 15:09:00.017667 | hotstack | ok: "artifacts"
2025-10-11 15:09:00.249934 | hotstack | ok: "docs"
2025-10-11 15:09:00.268995 | 
2025-10-11 15:09:00.269187 | LOOP [ensure-output-dirs : Ensure Zuul Output directories exist]
2025-10-11 15:09:00.577997 | hotstack | changed: "logs"
2025-10-11 15:09:00.831245 | hotstack | changed: "artifacts"
2025-10-11 15:09:01.071659 | hotstack | changed: "docs"
2025-10-11 15:09:01.119251 | 
2025-10-11 15:09:01.119414 | PLAY RECAP
2025-10-11 15:09:01.119503 | hotstack | ok: 22 changed: 9 unreachable: 0 failed: 0 skipped: 5 rescued: 0 ignored: 0
2025-10-11 15:09:01.119558 | localhost | ok: 6 changed: 2 unreachable: 0 failed: 0 skipped: 0 rescued: 0 ignored: 0
2025-10-11 15:09:01.119596 | 
2025-10-11 15:09:01.234537 | PRE-RUN END RESULT_NORMAL: [trusted : review.rdoproject.org/config/playbooks/base-minimal/pre.yaml@master]
2025-10-11 15:09:01.237126 | PRE-RUN START: [trusted : review.rdoproject.org/config/playbooks/hotstack/pre_stage1.yaml@master]
2025-10-11 15:09:01.772655 | 
2025-10-11 15:09:01.772789 | PLAY [Create necessary files to deploy hotstack scenario]
2025-10-11 15:09:01.792872 | 
2025-10-11 15:09:01.792947 | TASK [prepare-workspace : Start zuul_console daemon.]
2025-10-11 15:09:02.236307 | hotstack | ok
2025-10-11 15:09:02.249081 | 
2025-10-11 15:09:02.249243 | TASK [prepare-workspace : Synchronize src repos to workspace directory.]
2025-10-11 15:09:04.989574 | hotstack | Output suppressed because no_log was given
2025-10-11 15:09:05.009789 | 
2025-10-11 15:09:05.009971 | LOOP [add-authorized-keys : Enable access via build key on all nodes]
2025-10-11 15:09:05.503639 | hotstack | changed:
2025-10-11 15:09:05.503920 | hotstack | {
2025-10-11 15:09:05.503975 | hotstack |   "public_key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCINPOs6XsfMVQMqjmiduT3Bxhogy7MbAFOWlR+9LlKkTW1kV6glsCXY5svtggd2zRtc40UWalHaj9jC+yEkZdI35h2PkK+Sww52u4nczqgNpIFNt7Dm2xtYKbTolAV9jSeNkRlGMCq9EBzQlwiV0HkHx1CdNR/zzXbyNZLmOybFkXqR150H01kUhdv0v6AMWTDZjMEWO/PaWCOgzZhH9ztXjy+rwa7pdaI4+0Pb9Kib+d8/IrgNsLB/sCazqD+9/COI6AHqzTDS1aU9/gMQrUN1ronzMEQhzzOuoI8WeLqU++j+0dT2UWVPlQqoln+RYu7dUn/DSZDrsYenudT5+P"
2025-10-11 15:09:05.504018 | hotstack | }
2025-10-11 15:09:05.529000 | 
2025-10-11 15:09:05.529175 | TASK [Assert cloud name and cloud secret]
2025-10-11 15:09:05.598311 | hotstack | Output suppressed because no_log was given
2025-10-11 15:09:05.609020 | 
2025-10-11 15:09:05.609160 | TASK [Ensure openstack release repo is installed]
2025-10-11 15:09:10.551447 | hotstack | changed
2025-10-11 15:09:10.562542 | 
2025-10-11 15:09:10.562738 | TASK [Install openstack client packages]
2025-10-11 15:09:26.483836 | hotstack | changed
2025-10-11 15:09:26.494934 | 
2025-10-11 15:09:26.495056 | TASK [Ensure HotStack work dir]
2025-10-11 15:09:26.942146 | hotstack | changed
2025-10-11 15:09:26.953591 | 
2025-10-11 15:09:26.953752 | LOOP [Ensure etc directories]
2025-10-11 15:09:27.218611 | hotstack | changed: "/etc/openstack"
2025-10-11 15:09:27.218913 | hotstack | changed: All items complete
2025-10-11 15:09:27.218940 | 
2025-10-11 15:09:27.488661 | hotstack | changed: "/etc/hotstack"
2025-10-11 15:09:27.502928 | 
2025-10-11 15:09:27.503115 | TASK [Ensure .ssh dir exists]
2025-10-11 15:09:27.734992 | hotstack | Output suppressed because no_log was given
2025-10-11 15:09:27.746355 | 
2025-10-11 15:09:27.746481 | TASK [Ensure ssh keypair exists]
2025-10-11 15:09:38.306941 | hotstack | Output suppressed because no_log was given
2025-10-11 15:09:38.312662 | 
2025-10-11 15:09:38.312782 | TASK [Generate CA cert file]
2025-10-11 15:09:38.346954 | hotstack | skipping: Conditional result was False
2025-10-11 15:09:38.352505 | 
2025-10-11 15:09:38.352563 | TASK [Update CA certs]
2025-10-11 15:09:38.879838 | hotstack | skipping: Conditional result was False
2025-10-11 15:09:38.893509 | 
2025-10-11 15:09:38.893643 | TASK [Set _auth_type fact]
2025-10-11 15:09:38.951328 | hotstack | Output suppressed because no_log was given
2025-10-11 15:09:38.961251 | 
2025-10-11 15:09:38.961379 | TASK [Write clouds.yaml - user/password]
2025-10-11 15:09:40.201931 | hotstack | Output suppressed because no_log was given
2025-10-11 15:09:40.213512 | 
2025-10-11 15:09:40.213636 | TASK [Create application credential for sushy-emulator]
2025-10-11 15:09:43.265669 | hotstack | Output suppressed because no_log was given
2025-10-11 15:09:43.276176 | 
2025-10-11 15:09:43.276298 | TASK [Read application credential from file]
2025-10-11 15:09:43.644297 | hotstack | Output suppressed because no_log was given
2025-10-11 15:09:43.649494 | 
2025-10-11 15:09:43.649557 | TASK [Delete temporary application credential file]
2025-10-11 15:09:43.907877 | hotstack | Output suppressed because no_log was given
2025-10-11 15:09:43.918898 | 
2025-10-11 15:09:43.919053 | TASK [Assert application credential id and secret]
2025-10-11 15:09:44.007345 | hotstack | Output suppressed because no_log was given
2025-10-11 15:09:44.018149 | 
2025-10-11 15:09:44.018282 | TASK [Write new clouds.yaml with application credentials]
2025-10-11 15:09:44.972212 | hotstack | Output suppressed because no_log was given
2025-10-11 15:09:44.983033 | 
2025-10-11 15:09:44.983190 | TASK [Write hotstack cloud-secret.yaml]
2025-10-11 15:09:45.847034 | hotstack | Output suppressed because no_log was given
2025-10-11 15:09:45.858054 | 
2025-10-11 15:09:45.858208 | TASK [Write clouds.yaml with application credentials]
2025-10-11 15:09:45.896567 | 
2025-10-11 15:09:45.896893 | TASK [Write hotstack cloud-secret.yaml]
2025-10-11 15:09:45.990226 | 
2025-10-11 15:09:45.990363 | PLAY RECAP
2025-10-11 15:09:45.990415 | hotstack | ok: 18 changed: 11 unreachable: 0 failed: 0 skipped: 4 rescued: 0 ignored: 0
2025-10-11 15:09:45.990442 | 
2025-10-11 15:09:46.081363 | PRE-RUN END RESULT_NORMAL: [trusted : review.rdoproject.org/config/playbooks/hotstack/pre_stage1.yaml@master]
2025-10-11 15:09:46.083956 | PRE-RUN START: [trusted : review.rdoproject.org/config/playbooks/hotstack/pre_stage2.yaml@master]
2025-10-11 15:09:46.637367 | 
2025-10-11 15:09:46.637478 | PLAY [Create hotstack scenario]
2025-10-11 15:09:46.657933 | 
2025-10-11 15:09:46.658002 | TASK [Assert scenario is defined]
2025-10-11 15:09:46.730017 | hotstack | ok: All assertions passed
2025-10-11 15:09:46.736794 | 
2025-10-11 15:09:46.736890 | TASK [Fetch bootstrap vars from remote to zuul executor]
2025-10-11 15:09:47.260971 | hotstack | Output suppressed because no_log was given
2025-10-11 15:09:47.271590 | 
2025-10-11 15:09:47.271756 | TASK [Load bootstrap vars with template resolution]
2025-10-11 15:09:47.318406 | hotstack | Output suppressed because no_log was given
2025-10-11 15:09:47.329221 | 
2025-10-11 15:09:47.329350 | TASK [Fetch automation vars from remote to zuul executor]
2025-10-11 15:09:47.763422 | hotstack | Output suppressed because no_log was given
2025-10-11 15:09:47.773334 | 
2025-10-11 15:09:47.773462 | TASK [Load automation vars with template resolution]
2025-10-11 15:09:47.861285 | hotstack | Output suppressed because no_log was given
2025-10-11 15:09:47.871376 | 
2025-10-11 15:09:47.871502 | TASK [Slurp zuul-build-sshkey from file]
2025-10-11 15:09:48.255744 | hotstack | Output suppressed because no_log was given
2025-10-11 15:09:48.267370 | 
2025-10-11 15:09:48.267498 | TASK [Set up infrastructure (Heat stack)]
2025-10-11 15:09:48.295567 | hotstack | Output suppressed because no_log was given
2025-10-11 15:09:48.354852 | 
2025-10-11 15:09:48.354953 | TASK [dataplane_ssh_keys : Ensure Dataplane ssh keypair exists]
2025-10-11 15:09:49.012823 | hotstack | changed
2025-10-11 15:09:49.024182 | 
2025-10-11 15:09:49.024309 | TASK [dataplane_ssh_keys : Ensure Nova migration ssh keypair exists]
2025-10-11 15:09:49.443852 | hotstack | changed
2025-10-11 15:09:49.449742 | 
2025-10-11 15:09:49.449805 | TASK [dataplane_ssh_keys : Set SSH public key facts]
2025-10-11 15:09:49.495127 | hotstack | ok
2025-10-11 15:09:49.506893 | 
2025-10-11 15:09:49.507031 | TASK [dataplane_ssh_keys : Store SSH public key facts in file]
2025-10-11 15:09:50.497166 | hotstack | changed
2025-10-11 15:09:50.506571 | 
2025-10-11 15:09:50.506634 | TASK [heat_stack : Assert config is defined]
2025-10-11 15:09:50.610950 | hotstack | ok: All assertions passed
2025-10-11 15:09:50.619958 | 
2025-10-11 15:09:50.620055 | TASK [heat_stack : Get latest snapset and update stack parameters]
2025-10-11 15:09:50.667354 | hotstack | skipping: Conditional result was False
2025-10-11 15:09:50.674484 | 
2025-10-11 15:09:50.674557 | TASK [heat_stack : Debug latest snapset information]
2025-10-11 15:09:50.718517 | hotstack | skipping: Conditional result was False
2025-10-11 15:09:50.724299 | 
2025-10-11 15:09:50.724360 | TASK [heat_stack : Create stack]
2025-10-11 15:11:56.637727 | hotstack | changed
2025-10-11 15:11:56.655365 | 
2025-10-11 15:11:56.655492 | TASK [heat_stack : Wait for stack to complete]
2025-10-11 15:11:58.317833 | hotstack | ok
2025-10-11 15:11:58.329395 | 
2025-10-11 15:11:58.329525 | TASK [heat_stack : Debug - stack_info]
2025-10-11 15:11:58.356636 | hotstack | skipping: Conditional result was False
2025-10-11 15:11:58.369143 | 
2025-10-11 15:11:58.369277 | TASK [heat_stack : Stop if stack not complete]
2025-10-11 15:11:58.395956 | hotstack | skipping: Conditional result was False
2025-10-11 15:11:58.407674 | 
2025-10-11 15:11:58.407857 | TASK [heat_stack : Get stack outputs]
2025-10-11 15:12:00.308118 | hotstack | changed
2025-10-11 15:12:00.324327 | 
2025-10-11 15:12:00.324460 | TASK [heat_stack : Set stack outputs facts]
2025-10-11 15:12:00.372758 | hotstack | ok
2025-10-11 15:12:00.388532 | 
2025-10-11 15:12:00.388664 | TASK [heat_stack : Store stack outputs in file]
2025-10-11 15:12:01.362163 | hotstack | changed
2025-10-11 15:12:01.383057 | 
2025-10-11 15:12:01.383205 | TASK [Fetch stack outputs from remote to zuul executor]
2025-10-11 15:12:01.797591 | hotstack | Output suppressed because no_log was given
2025-10-11 15:12:01.807420 | 
2025-10-11 15:12:01.807581 | TASK [Load stack output vars with template resolution]
2025-10-11 15:12:01.858844 | hotstack | Output suppressed because no_log was given
2025-10-11 15:12:01.871527 | 
2025-10-11 15:12:01.871662 | TASK [Fetch cloud secrets from remote to zuul executor]
2025-10-11 15:12:02.258832 | hotstack | Output suppressed because no_log was given
2025-10-11 15:12:02.265419 | 
2025-10-11 15:12:02.265501 | TASK [Load hotstack cloud secret vars with template resolution]
2025-10-11 15:12:02.308522 | hotstack | Output suppressed because no_log was given
2025-10-11 15:12:02.315538 | 
2025-10-11 15:12:02.315619 | TASK [Add Hotstack controller-0 to ansible inventory]
2025-10-11 15:12:02.357848 | hotstack | Output suppressed because no_log was given
2025-10-11 15:12:02.365558 | 
2025-10-11 15:12:02.365643 | TASK [Wait for Hotstack controller-0 to be ready]
2025-10-11 15:12:08.634330 | hotstack -> 38.102.83.64 | Output suppressed because no_log was given
2025-10-11 15:12:08.645786 | 
2025-10-11 15:12:08.645917 | TASK [Start Zuul console on Hotstack controller-0]
2025-10-11 15:12:08.673494 | hotstack | Output suppressed because no_log was given
2025-10-11 15:12:08.714980 | 
2025-10-11 15:12:08.715140 | TASK [start-zuul-console : Start zuul_console daemon.]
2025-10-11 15:12:09.238564 | hotstack -> 38.102.83.64 | ok
2025-10-11 15:12:09.252943 | 
2025-10-11 15:12:09.253038 | TASK [Prepare the controller node]
2025-10-11 15:12:09.275388 | hotstack | Output suppressed because no_log was given
2025-10-11 15:12:09.353364 | 
2025-10-11 15:12:09.353488 | TASK [controller : Assert config is defined]
2025-10-11 15:12:09.432802 | hotstack | ok: All assertions passed
2025-10-11 15:12:09.437572 | 
2025-10-11 15:12:09.437633 | TASK [controller : Add controller-0 to the Ansible inventory]
2025-10-11 15:12:09.466789 | hotstack | ok
2025-10-11 15:12:09.471709 | 
2025-10-11 15:12:09.471769 | TASK [controller : Read Dataplane SSH private key content]
2025-10-11 15:12:09.719141 | hotstack | ok
2025-10-11 15:12:09.731441 | 
2025-10-11 15:12:09.731574 | TASK [controller : Read Nova migration SSH private key content]
2025-10-11 15:12:10.012961 | hotstack | ok
2025-10-11 15:12:10.023770 | 
2025-10-11 15:12:10.023905 | TASK [controller : Wait for controller-0 to be ready]
2025-10-11 15:12:10.621243 | hotstack -> 38.102.83.64 | ok
2025-10-11 15:12:10.633709 | 
2025-10-11 15:12:10.633842 | LOOP [controller : Ensure directories exists]
2025-10-11 15:12:11.142106 | hotstack -> 38.102.83.64 | changed: "/home/zuul/data"
2025-10-11 15:12:11.446603 | hotstack -> 38.102.83.64 | changed: "/home/zuul/bin"
2025-10-11 15:12:11.463541 | 
2025-10-11 15:12:11.463721 | LOOP [controller : Add common functions to the bin_dir]
2025-10-11 15:12:12.292527 | hotstack -> 38.102.83.64 | changed: "bin/hotstack-common-funcs.sh"
2025-10-11 15:12:12.292978 | hotstack -> 38.102.83.64 | changed: All items complete
2025-10-11 15:12:12.293038 | 
2025-10-11 15:12:12.307289 | 
2025-10-11 15:12:12.307427 | LOOP [controller : Add bin utilities to the bin_dir]
2025-10-11 15:12:13.112323 | hotstack -> 38.102.83.64 | changed: "bin/hotstack-approve-install-plan"
2025-10-11 15:12:13.804364 | hotstack -> 38.102.83.64 | changed: "bin/hotstack-leader-election-tune"
2025-10-11 15:12:14.515140 | hotstack -> 38.102.83.64 | changed: "bin/hotstack-nova-discover-hosts"
2025-10-11 15:12:15.206906 | hotstack -> 38.102.83.64 | changed: "bin/hotstack-snapset"
2025-10-11 15:12:15.895982 | hotstack -> 38.102.83.64 | changed: "bin/hotstack-openstack-version-patch"
2025-10-11 15:12:16.590661 | hotstack -> 38.102.83.64 | changed: "bin/hotstack-wait-for-bmh"
2025-10-11 15:12:16.608293 | 
2025-10-11 15:12:16.608432 | TASK [controller : Write ansible inventory to file on controller-0]
2025-10-11 15:12:16.696486 | hotstack | skipping: Conditional result was False
2025-10-11 15:12:16.709584 | 
2025-10-11 15:12:16.709783 | TASK [controller : Copy Dataplane ssh private key to controller]
2025-10-11 15:12:17.502790 | hotstack -> 38.102.83.64 | changed
2025-10-11 15:12:17.515312 | 
2025-10-11 15:12:17.515445 | TASK [controller : Copy Dataplane ssh public key to controller]
2025-10-11 15:12:18.246670 | hotstack -> 38.102.83.64 | changed
2025-10-11 15:12:18.259044 | 
2025-10-11 15:12:18.259175 | TASK [controller : Copy Nova migrate ssh private key to controller]
2025-10-11 15:12:19.025177 | hotstack -> 38.102.83.64 | changed
2025-10-11 15:12:19.037451 | 
2025-10-11 15:12:19.037582 | TASK [controller : Copy Nova migrate ssh public key to controller]
2025-10-11 15:12:19.839012 | hotstack -> 38.102.83.64 | changed
2025-10-11 15:12:19.852325 | 
2025-10-11 15:12:19.852468 | TASK [controller : Copy dataplane authorized keys to controller]
2025-10-11 15:12:20.614282 | hotstack -> 38.102.83.64 | changed
2025-10-11 15:12:20.627836 | 
2025-10-11 15:12:20.627973 | TASK [controller : Ensure cloud config directory exists]
2025-10-11 15:12:21.015606 | hotstack -> 38.102.83.64 | changed
2025-10-11 15:12:21.029308 | 
2025-10-11 15:12:21.029442 | TASK [controller : Template the clouds.yaml]
2025-10-11 15:12:21.875141 | hotstack -> 38.102.83.64 | Output suppressed because no_log was given
2025-10-11 15:12:21.887439 | 
2025-10-11 15:12:21.887573 | TASK [controller : Write Ironic nodes YAML]
2025-10-11 15:12:21.934665 | hotstack | skipping: Conditional result was False
2025-10-11 15:12:21.956241 | 
2025-10-11 15:12:21.956369 | TASK [Install Openshift]
2025-10-11 15:12:21.983643 | hotstack | Output suppressed because no_log was given
2025-10-11 15:12:22.042325 | 
2025-10-11 15:12:22.042392 | TASK [ocp_agent_installer : Assert config is defined]
2025-10-11 15:12:22.127382 | hotstack -> 38.102.83.64 | ok: All assertions passed
2025-10-11 15:12:22.139557 | 
2025-10-11 15:12:22.139720 | TASK [ocp_agent_installer : Install using agent installer]
2025-10-11 15:12:22.207321 | hotstack | ok
2025-10-11 15:12:22.237128 | hotstack | included: /var/lib/zuul/builds/918b704357334bf18129d9b4a9d53906/trusted/project_3/github.com/openstack-k8s-operators/hotstack/roles/ocp_agent_installer/tasks/_install.yml
2025-10-11 15:12:22.248102 | 
2025-10-11 15:12:22.248185 | TASK [ocp_agent_installer : Assert config is defined]
2025-10-11 15:12:22.334032 | hotstack -> 38.102.83.64 | ok: All assertions passed
2025-10-11 15:12:22.341713 | 
2025-10-11 15:12:22.341792 | LOOP [ocp_agent_installer : Ensure directory exists]
2025-10-11 15:12:22.740258 | hotstack -> 38.102.83.64 | ok: "/home/zuul/bin"
2025-10-11 15:12:22.740767 | hotstack -> 38.102.83.64 | changed: All items complete
2025-10-11 15:12:22.740836 | 
2025-10-11 15:12:23.033330 | hotstack -> 38.102.83.64 | changed: "/home/zuul/.kube"
2025-10-11 15:12:23.327426 | hotstack -> 38.102.83.64 | changed: "/home/zuul/ocp-cluster"
2025-10-11 15:12:23.618856 | hotstack -> 38.102.83.64 | changed: "/home/zuul/ocp-cluster/openshift"
2025-10-11 15:12:23.911861 | hotstack -> 38.102.83.64 | changed: "/home/zuul/agent-installer"
2025-10-11 15:12:24.229773 | hotstack -> 38.102.83.64 | changed: "/home/zuul/cluster-custom-config/"
2025-10-11 15:12:24.521315 | hotstack -> 38.102.83.64 | changed: "/home/zuul/cluster-custom-config//butane"
2025-10-11 15:12:24.809819 | hotstack -> 38.102.83.64 | changed: "/home/zuul/cluster-custom-config//machine-configs"
2025-10-11 15:12:25.095702 | hotstack -> 38.102.83.64 | changed: "/home/zuul/cluster-custom-config//config-assets"
2025-10-11 15:12:25.120521 | 
2025-10-11 15:12:25.120757 | TASK [ocp_agent_installer : Install package requirements for agent installer]
2025-10-11 15:12:41.351984 | hotstack -> 38.102.83.64 | ok: Nothing to do
2025-10-11 15:12:41.364614 | 
2025-10-11 15:12:41.364789 | TASK [ocp_agent_installer : Run tasks/install_client.yml]
2025-10-11 15:12:41.402461 | hotstack | ok
2025-10-11 15:12:41.418265 | hotstack | included: /var/lib/zuul/builds/918b704357334bf18129d9b4a9d53906/trusted/project_3/github.com/openstack-k8s-operators/hotstack/roles/ocp_agent_installer/tasks/install_client.yml
2025-10-11 15:12:41.437742 | 
2025-10-11 15:12:41.437826 | TASK [ocp_agent_installer : Download the client]
2025-10-11 15:12:43.721597 | hotstack -> 38.102.83.64 | changed: OK (73347743 bytes)
2025-10-11 15:12:43.735600 | 
2025-10-11 15:12:43.735786 | TASK [ocp_agent_installer : Extract client to /home/zuul/bin]
2025-10-11 15:12:49.612142 | hotstack -> 38.102.83.64 | changed
2025-10-11 15:12:49.625858 | 
2025-10-11 15:12:49.625993 | TASK [ocp_agent_installer : Configure bash completion]
2025-10-11 15:12:50.660311 | hotstack -> 38.102.83.64 | changed
2025-10-11 15:12:50.673635 | 
2025-10-11 15:12:50.673815 | TASK [ocp_agent_installer : Run tasks/install_installer.yml]
2025-10-11 15:12:50.711234 | hotstack | ok
2025-10-11 15:12:50.725305 | hotstack | included: /var/lib/zuul/builds/918b704357334bf18129d9b4a9d53906/trusted/project_3/github.com/openstack-k8s-operators/hotstack/roles/ocp_agent_installer/tasks/install_installer.yml
2025-10-11 15:12:50.748624 | 
2025-10-11 15:12:50.748736 | TASK [ocp_agent_installer : Download the installer]
2025-10-11 15:12:58.219443 | hotstack -> 38.102.83.64 | changed: OK (437684516 bytes)
2025-10-11 15:12:58.232801 | 
2025-10-11 15:12:58.232935 | TASK [ocp_agent_installer : Extract installer to /home/zuul/bin]
2025-10-11 15:13:13.381278 | hotstack -> 38.102.83.64 | changed
2025-10-11 15:13:13.395002 | 
2025-10-11 15:13:13.395144 | TASK [ocp_agent_installer : Write OCP install-config.yaml]
2025-10-11 15:13:14.255877 | hotstack -> 38.102.83.64 | changed
2025-10-11 15:13:14.268740 | 
2025-10-11 15:13:14.268900 | TASK [ocp_agent_installer : Replace pull secret in install-config.yaml]
2025-10-11 15:13:14.774484 | hotstack -> 38.102.83.64 | Output suppressed because no_log was given
2025-10-11 15:13:14.786763 | 
2025-10-11 15:13:14.786898 | TASK [ocp_agent_installer : Write OCP agent-config.yaml]
2025-10-11 15:13:15.587864 | hotstack -> 38.102.83.64 | changed
2025-10-11 15:13:15.602471 | 
2025-10-11 15:13:15.602602 | TASK [ocp_agent_installer : Include tasks to generate machine configs]
2025-10-11 15:13:15.643277 | hotstack | ok
2025-10-11 15:13:15.662165 | hotstack | included: /var/lib/zuul/builds/918b704357334bf18129d9b4a9d53906/trusted/project_3/github.com/openstack-k8s-operators/hotstack/roles/ocp_agent_installer/tasks/machine_configs.yml
2025-10-11 15:13:15.681708 | 
2025-10-11 15:13:15.681813 | LOOP [ocp_agent_installer : Template butane config for net.ifnames]
2025-10-11 15:13:16.513568 | hotstack -> 38.102.83.64 | changed: "master"
2025-10-11 15:13:16.530501 | 
2025-10-11 15:13:16.530643 | LOOP [ocp_agent_installer : Generate MachineConfig for iscsi]
2025-10-11 15:13:17.021412 | hotstack -> 38.102.83.64 | changed
2025-10-11 15:13:17.021883 | hotstack -> 38.102.83.64 | changed: All items complete
2025-10-11 15:13:17.021946 | 
2025-10-11 15:13:17.038100 | 
2025-10-11 15:13:17.038238 | LOOP [ocp_agent_installer : Template butane config for iscsi]
2025-10-11 15:13:17.839878 | hotstack -> 38.102.83.64 | changed: "master"
2025-10-11 15:13:17.854357 | 
2025-10-11 15:13:17.854499 | LOOP [ocp_agent_installer : Generate MachineConfig for iscsi]
2025-10-11 15:13:18.288730 | hotstack -> 38.102.83.64 | changed
2025-10-11 15:13:18.289115 | hotstack -> 38.102.83.64 | changed: All items complete
2025-10-11 15:13:18.289172 | 
2025-10-11 15:13:18.303032 | 
2025-10-11 15:13:18.303190 | LOOP [ocp_agent_installer : Template butane config for multipath]
2025-10-11 15:13:19.099150 | hotstack -> 38.102.83.64 | changed: "master"
2025-10-11 15:13:19.113780 | 
2025-10-11 15:13:19.113977 | LOOP [ocp_agent_installer : Generate MachineConfig for multipath]
2025-10-11 15:13:19.538798 | hotstack -> 38.102.83.64 | changed
2025-10-11 15:13:19.539211 | hotstack -> 38.102.83.64 | changed: All items complete
2025-10-11 15:13:19.539285 | 
2025-10-11 15:13:19.553534 | 
2025-10-11 15:13:19.553748 | LOOP [ocp_agent_installer : Template butane config for LVM cinder-volumes]
2025-10-11 15:13:20.415297 | hotstack -> 38.102.83.64 | changed: "master"
2025-10-11 15:13:20.433004 | 
2025-10-11 15:13:20.433205 | LOOP [ocp_agent_installer : Generate MachineConfig for multipath]
2025-10-11 15:13:20.873185 | hotstack -> 38.102.83.64 | changed
2025-10-11 15:13:20.873441 | hotstack -> 38.102.83.64 | changed: All items complete
2025-10-11 15:13:20.873469 | 
2025-10-11 15:13:20.890478 | 
2025-10-11 15:13:20.890648 | TASK [ocp_agent_installer : Include tasks to generate config assets]
2025-10-11 15:13:20.928611 | hotstack | ok
2025-10-11 15:13:20.948570 | hotstack | included: /var/lib/zuul/builds/918b704357334bf18129d9b4a9d53906/trusted/project_3/github.com/openstack-k8s-operators/hotstack/roles/ocp_agent_installer/tasks/config_assets.yml
2025-10-11 15:13:20.982254 | 
2025-10-11 15:13:20.982343 | TASK [ocp_agent_installer : Template ovn-k8s customization]
2025-10-11 15:13:21.922301 | hotstack -> 38.102.83.64 | changed
2025-10-11 15:13:21.929790 | 
2025-10-11 15:13:21.929872 | TASK [ocp_agent_installer : Copy Etcd customization]
2025-10-11 15:13:22.738053 | hotstack -> 38.102.83.64 | changed
2025-10-11 15:13:22.751439 | 
2025-10-11 15:13:22.751572 | TASK [ocp_agent_installer : Template ImageContentSourcePolicy customization]
2025-10-11 15:13:22.819369 | hotstack | skipping: Conditional result was False
2025-10-11 15:13:22.831846 | 
2025-10-11 15:13:22.831984 | TASK [ocp_agent_installer : Initialize _ocp_additional_trusted_ca_map fact]
2025-10-11 15:13:22.890063 | hotstack | skipping: Conditional result was False
2025-10-11 15:13:22.903023 | 
2025-10-11 15:13:22.903161 | LOOP [ocp_agent_installer : Append to _ocp_additional_trusted_ca_map fact]
2025-10-11 15:13:22.959392 | hotstack | skipping: Conditional result was False
2025-10-11 15:13:22.985531 | 
2025-10-11 15:13:22.985723 | TASK [ocp_agent_installer : Template additional CA config map]
2025-10-11 15:13:23.033100 | hotstack | skipping: Conditional result was False
2025-10-11 15:13:23.046607 | 
2025-10-11 15:13:23.046778 | TASK [ocp_agent_installer : Copy additional CA config image]
2025-10-11 15:13:23.093787 | hotstack | skipping: Conditional result was False
2025-10-11 15:13:23.106026 | 
2025-10-11 15:13:23.106156 | TASK [ocp_agent_installer : Copy machine_configs to manifests dir]
2025-10-11 15:13:23.599436 | hotstack -> 38.102.83.64 | changed
2025-10-11 15:13:23.605647 | 
2025-10-11 15:13:23.605727 | TASK [ocp_agent_installer : Copy config assets to manifests dir]
2025-10-11 15:13:24.087972 | hotstack -> 38.102.83.64 | changed
2025-10-11 15:13:24.101828 | 
2025-10-11 15:13:24.101965 | TASK [ocp_agent_installer : Run tasks/pxe_assets.yml]
2025-10-11 15:13:24.153464 | hotstack | ok
2025-10-11 15:13:24.171197 | hotstack | included: /var/lib/zuul/builds/918b704357334bf18129d9b4a9d53906/trusted/project_3/github.com/openstack-k8s-operators/hotstack/roles/ocp_agent_installer/tasks/pxe_assets.yml
2025-10-11 15:13:24.202173 | 
2025-10-11 15:13:24.202260 | TASK [ocp_agent_installer : Create the boot-artifacts directory]
2025-10-11 15:13:24.601874 | hotstack -> 38.102.83.64 | changed
2025-10-11 15:13:24.610421 | 
2025-10-11 15:13:24.610542 | TASK [ocp_agent_installer : Create PXE assets]
2025-10-11 15:13:26.556672 | controller-0 | level=warning msg=Compute[0].Hyperthreading: Disabled is ignored
2025-10-11 15:13:27.505570 | controller-0 | level=warning msg=ControlPlane.Hyperthreading: Disabled is ignored
2025-10-11 15:13:27.505614 | controller-0 | level=info msg=Configuration has 3 master replicas and 0 worker replicas
2025-10-11 15:13:27.505630 | controller-0 | level=info msg=The rendezvous host IP (node0 IP) is 192.168.34.10
2025-10-11 15:13:28.165977 | controller-0 | level=info msg=Extracting base ISO from release payload
2025-10-11 15:13:55.114807 | controller-0 | level=info msg=Base ISO obtained from release and cached at [/home/zuul/.cache/agent/image_cache/coreos-x86_64.iso]
2025-10-11 15:14:43.997611 | controller-0 | level=info msg=Consuming Agent Config from target directory
2025-10-11 15:14:43.998021 | controller-0 | level=info msg=Consuming Install Config from target directory
2025-10-11 15:14:44.595646 | controller-0 | level=info msg=Consuming Extra Manifests from target directory
2025-10-11 15:14:44.595705 | controller-0 | level=info msg=Created iPXE script agent.x86_64.ipxe in boot-artifacts directory
2025-10-11 15:14:44.595773 | controller-0 | level=info msg=PXE boot artifacts created in: boot-artifacts
2025-10-11 15:14:44.595856 | controller-0 | level=info msg=Kernel parameters for PXE boot: rw ignition.firstboot ignition.platform.id=metal
2025-10-11 15:14:45.217397 | hotstack -> 38.102.83.64 | changed
2025-10-11 15:14:45.225959 | 
2025-10-11 15:14:45.226042 | TASK [ocp_agent_installer : Set serial console in ipxe]
2025-10-11 15:14:45.883038 | hotstack -> 38.102.83.64 | changed: line replaced
2025-10-11 15:14:45.889347 | 
2025-10-11 15:14:45.889408 | TASK [ocp_agent_installer : Disable net.ifnames]
2025-10-11 15:14:46.453470 | hotstack -> 38.102.83.64 | changed: line replaced
2025-10-11 15:14:46.467853 | 
2025-10-11 15:14:46.467988 | LOOP [ocp_agent_installer : Copy boot-artifacts to the web server - (ocp_agent_installer_boot_artifacts_dir)]
2025-10-11 15:14:47.421210 | hotstack -> 38.102.83.64 | changed: "agent.x86_64-vmlinuz"
2025-10-11 15:14:47.421628 | 
2025-10-11 15:14:48.831084 | hotstack -> 38.102.83.64 | changed: "agent.x86_64-initrd.img"
2025-10-11 15:15:12.494168 | hotstack -> 38.102.83.64 | changed: "agent.x86_64-rootfs.img"
2025-10-11 15:15:12.973838 | hotstack -> 38.102.83.64 | changed: "agent.x86_64.ipxe"
2025-10-11 15:15:12.983621 | 
2025-10-11 15:15:12.983779 | TASK [ocp_agent_installer : Run tasks/iso_assets.yml]
2025-10-11 15:15:13.018599 | hotstack | skipping: Conditional result was False
2025-10-11 15:15:13.024432 | 
2025-10-11 15:15:13.024495 | TASK [ocp_agent_installer : Copy auth/kubeconfig to ~/.kube/config]
2025-10-11 15:15:13.577988 | hotstack -> 38.102.83.64 | changed
2025-10-11 15:15:13.590869 | 
2025-10-11 15:15:13.591007 | TASK [ocp_agent_installer : Wait for bootstrap-complete]
2025-10-11 15:15:54.157042 | controller-0 | level=info msg=Waiting for cluster install to initialize. Sleeping for 30 seconds
2025-10-11 15:16:36.177492 | controller-0 | level=info msg=Waiting for cluster install to initialize. Sleeping for 30 seconds
2025-10-11 15:17:18.201034 | controller-0 | level=info msg=Cannot access Rendezvous Host. There may be a network configuration problem, check console for additional info
2025-10-11 15:18:00.224326 | controller-0 | level=info msg=Cannot access Rendezvous Host. There may be a network configuration problem, check console for additional info
2025-10-11 15:18:42.268056 | controller-0 | level=info msg=Cluster is not ready for install. Check validations
2025-10-11 15:18:42.268506 | controller-0 | level=warning msg=Cluster validation: The cluster has hosts that are not ready to install.
2025-10-11 15:18:42.269765 | controller-0 | level=warning msg=Host master-0 validation: No connectivity to the majority of hosts in the cluster
2025-10-11 15:18:42.269909 | controller-0 | level=warning msg=Host master-0 validation: Host couldn't synchronize with any NTP server
2025-10-11 15:18:42.270149 | controller-0 | level=warning msg=Host master-0 validation: Error while evaluating DNS resolution on this host
2025-10-11 15:18:42.270294 | controller-0 | level=warning msg=Host master-0 validation: Error while evaluating DNS resolution on this host
2025-10-11 15:18:42.270376 | controller-0 | level=warning msg=Host master-0 validation: Error while evaluating DNS resolution on this host
2025-10-11 15:18:42.271584 | controller-0 | level=warning msg=Host master-2 validation: No connectivity to the majority of hosts in the cluster
2025-10-11 15:18:42.271677 | controller-0 | level=warning msg=Host master-2 validation: Host couldn't synchronize with any NTP server
2025-10-11 15:18:42.271821 | controller-0 | level=warning msg=Host master-2 validation: Error while evaluating DNS resolution on this host
2025-10-11 15:18:42.271883 | controller-0 | level=warning msg=Host master-2 validation: Error while evaluating DNS resolution on this host
2025-10-11 15:18:42.271942 | controller-0 | level=warning msg=Host master-2 validation: Error while evaluating DNS resolution on this host
2025-10-11 15:18:42.272784 | controller-0 | level=warning msg=Host master-1 validation: No connectivity to the majority of hosts in the cluster
2025-10-11 15:18:42.272879 | controller-0 | level=warning msg=Host master-1 validation: Host couldn't synchronize with any NTP server
2025-10-11 15:18:42.273024 | controller-0 | level=warning msg=Host master-1 validation: Error while evaluating DNS resolution on this host
2025-10-11 15:18:42.273100 | controller-0 | level=warning msg=Host master-1 validation: Error while evaluating DNS resolution on this host
2025-10-11 15:18:42.273163 | controller-0 | level=warning msg=Host master-1 validation: Error while evaluating DNS resolution on this host
2025-10-11 15:18:42.279408 | controller-0 | level=warning msg=Host master-0: updated status from discovering to insufficient (Host cannot be installed due to following failing validation(s): Host couldn't synchronize with any NTP server ; No connectivity to the majority of hosts in the cluster)
2025-10-11 15:19:42.478274 | controller-0 | level=info msg=Host master-0 validation: Host NTP is synced
2025-10-11 15:19:42.478481 | controller-0 | level=info msg=Host master-0 validation: Domain name resolution for the api.ocp.openstack.lab domain was successful or not required
2025-10-11 15:19:42.478564 | controller-0 | level=info msg=Host master-0 validation: Domain name resolution for the api-int.ocp.openstack.lab domain was successful or not required
2025-10-11 15:19:42.478656 | controller-0 | level=info msg=Host master-0 validation: Domain name resolution for the *.apps.ocp.openstack.lab domain was successful or not required
2025-10-11 15:19:42.479053 | controller-0 | level=info msg=Host master-2 validation: Host NTP is synced
2025-10-11 15:19:42.479138 | controller-0 | level=info msg=Host master-2 validation: Domain name resolution for the api.ocp.openstack.lab domain was successful or not required
2025-10-11 15:19:42.479265 | controller-0 | level=info msg=Host master-2 validation: Domain name resolution for the api-int.ocp.openstack.lab domain was successful or not required
2025-10-11 15:19:42.479373 | controller-0 | level=info msg=Host master-2 validation: Domain name resolution for the *.apps.ocp.openstack.lab domain was successful or not required
2025-10-11 15:19:42.479739 | controller-0 | level=info msg=Host master-1 validation: Host NTP is synced
2025-10-11 15:19:42.479862 | controller-0 | level=info msg=Host master-1 validation: Domain name resolution for the api.ocp.openstack.lab domain was successful or not required
2025-10-11 15:19:42.479974 | controller-0 | level=info msg=Host master-1 validation: Domain name resolution for the api-int.ocp.openstack.lab domain was successful or not required
2025-10-11 15:19:42.480037 | controller-0 | level=info msg=Host master-1 validation: Domain name resolution for the *.apps.ocp.openstack.lab domain was successful or not required
2025-10-11 15:19:42.483575 | controller-0 | level=info msg=Host master-1: validation 'ntp-synced' is now fixed
2025-10-11 15:19:54.521564 | controller-0 | level=info msg=Host master-1 validation: Host has connectivity to the majority of hosts in the cluster
2025-10-11 15:19:54.525507 | controller-0 | level=info msg=Host master-1: updated status from insufficient to known (Host is ready to be installed)
2025-10-11 15:20:06.563714 | controller-0 | level=info msg=Cluster is ready for install
2025-10-11 15:20:06.564001 | controller-0 | level=info msg=Cluster validation: All hosts in the cluster are ready to install.
2025-10-11 15:20:06.564287 | controller-0 | level=info msg=Host master-2 validation: Host has connectivity to the majority of hosts in the cluster
2025-10-11 15:20:06.564755 | controller-0 | level=info msg=Host master-0 validation: Host has connectivity to the majority of hosts in the cluster
2025-10-11 15:20:06.568661 | controller-0 | level=info msg=Host master-2: updated status from insufficient to known (Host is ready to be installed)
2025-10-11 15:20:18.603229 | controller-0 | level=info msg=Preparing cluster for installation
2025-10-11 15:20:18.608732 | controller-0 | level=info msg=Host master-0: updated status from known to preparing-for-installation (Host finished successfully to prepare for installation)
2025-10-11 15:20:54.743124 | controller-0 | level=info msg=Host master-0: updated status from preparing-for-installation to preparing-successful (Host finished successfully to prepare for installation)
2025-10-11 15:39:58.716450 | controller-0 | level=warning msg=Host's disk /dev/disk/by-path/pci-0000:00:06.0 is slower than the supported speed, and may cause degraded cluster performance (fdatasync duration: 34 ms)
2025-10-11 15:40:10.757927 | controller-0 | level=info msg=Host master-1: updated status from preparing-for-installation to preparing-successful (Host finished successfully to prepare for installation)
2025-10-11 15:40:22.794767 | controller-0 | level=info msg=Cluster installation in progress
2025-10-11 15:40:22.802068 | controller-0 | level=info msg=Host master-2: updated status from preparing-successful to installing (Installation is in progress)
2025-10-11 15:40:34.847262 | controller-0 | level=info msg=Host: master-2, reached installation stage Writing image to disk
2025-10-11 15:40:46.895975 | controller-0 | level=info msg=Host: master-2, reached installation stage Writing image to disk: 38%
2025-10-11 15:40:58.943782 | controller-0 | level=info msg=Host: master-0, reached installation stage Writing image to disk: 64%
2025-10-11 15:41:11.002007 | controller-0 | level=info msg=Host: master-0, reached installation stage Writing image to disk: 100%
2025-10-11 15:41:58.172260 | controller-0 | level=info msg=Bootstrap Kube API Initialized
2025-10-11 15:42:28.639968 | controller-0 | level=info msg=Host: master-2, reached installation stage Rebooting
2025-10-11 15:46:14.076369 | controller-0 | level=info msg=Host: master-2, reached installation stage Joined
2025-10-11 15:46:22.194486 | controller-0 | level=info msg=Host: master-2, reached installation stage Done
2025-10-11 15:46:30.302360 | controller-0 | level=info msg=Node master-2 has been rebooted 1 times before completing installation
2025-10-11 16:15:03.369135 | controller-0 | level=warning msg=Host master-1: updated status from installing-in-progress to error (Host failed to install because its installation stage Writing image to disk did not sufficiently progress in the last 30m0s.)
2025-10-11 16:15:13.510220 | controller-0 | level=info msg=Cluster has hosts in error
2025-10-11 16:15:13.510337 | controller-0 | level=info msg=cluster has stopped installing... working to recover installation
2025-10-11 16:15:14.164163 | controller-0 | level=error msg=Cluster operator authentication Degraded is True with IngressStateEndpoints_MissingSubsets::OAuthServerServiceEndpointAccessibleController_SyncError::OAuthServerServiceEndpointsEndpointAccessibleController_SyncError: IngressStateEndpointsDegraded: No subsets found for the endpoints of oauth-server
2025-10-11 16:15:14.164268 | controller-0 | level=error msg=OAuthServerServiceEndpointAccessibleControllerDegraded: Get "https://172.30.254.232:443/healthz": dial tcp 172.30.254.232:443: connect: connection refused
2025-10-11 16:15:14.164301 | controller-0 | level=error msg=OAuthServerServiceEndpointsEndpointAccessibleControllerDegraded: oauth service endpoints are not ready
2025-10-11 16:15:14.164399 | controller-0 | level=info msg=Cluster operator authentication Available is False with APIServices_PreconditionNotReady::OAuthServerServiceEndpointAccessibleController_EndpointUnavailable::OAuthServerServiceEndpointsEndpointAccessibleController_ResourceNotFound: APIServicesAvailable: PreconditionNotReady
2025-10-11 16:15:14.164429 | controller-0 | level=info msg=OAuthServerServiceEndpointAccessibleControllerAvailable: Get "https://172.30.254.232:443/healthz": dial tcp 172.30.254.232:443: connect: connection refused
2025-10-11 16:15:14.164453 | controller-0 | level=info msg=OAuthServerServiceEndpointsEndpointAccessibleControllerAvailable: endpoints "oauth-openshift" not found
2025-10-11 16:15:14.164508 | controller-0 | level=info msg=Cluster operator authentication EvaluationConditionsDetected is Unknown with NoData:
2025-10-11 16:15:14.164554 | controller-0 | level=info msg=Cluster operator baremetal Disabled is False with :
2025-10-11 16:15:14.164598 | controller-0 | level=info msg=Cluster operator cloud-controller-manager CloudConfigControllerAvailable is True with AsExpected: Cloud Config Controller works as expected
2025-10-11 16:15:14.164641 | controller-0 | level=info msg=Cluster operator cloud-controller-manager CloudConfigControllerDegraded is False with AsExpected: Cloud Config Controller works as expected
2025-10-11 16:15:14.164685 | controller-0 | level=info msg=Cluster operator cloud-controller-manager TrustedCABundleControllerControllerAvailable is True with AsExpected: Trusted CA Bundle Controller works as expected
2025-10-11 16:15:14.164726 | controller-0 | level=info msg=Cluster operator cloud-controller-manager TrustedCABundleControllerControllerDegraded is False with AsExpected: Trusted CA Bundle Controller works as expected
2025-10-11 16:15:14.164777 | controller-0 | level=info msg=Cluster operator config-operator EvaluationConditionsDetected is Unknown with NoData:
2025-10-11 16:15:14.164821 | controller-0 | level=info msg=Cluster operator csi-snapshot-controller EvaluationConditionsDetected is Unknown with NoData:
2025-10-11 16:15:14.164861 | controller-0 | level=error msg=Cluster operator etcd Degraded is True with RevisionController_SyncError: RevisionControllerDegraded: etcd cluster has quorum of 2 which is not fault tolerant: [{Member:ID:1222771942322075303 name:"master-2" peerURLs:"https://192.168.34.12:2380" clientURLs:"https://192.168.34.12:2379"  Healthy:true Took:2.736127ms Error:<nil>} {Member:ID:14813819669272913224 name:"etcd-bootstrap" peerURLs:"https://192.168.34.10:2380" clientURLs:"https://192.168.34.10:2379"  Healthy:true Took:3.418867ms Error:<nil>}]
2025-10-11 16:15:14.164916 | controller-0 | level=info msg=Cluster operator etcd EvaluationConditionsDetected is Unknown with NoData:
2025-10-11 16:15:14.164960 | controller-0 | level=info msg=Cluster operator ingress Available is False with IngressUnavailable: The "default" ingress controller reports Available=False: IngressControllerUnavailable: One or more status conditions indicate unavailable: DeploymentAvailable=False (DeploymentUnavailable: The deployment has Available status condition set to False (reason: MinimumReplicasUnavailable) with message: Deployment does not have minimum availability.)
2025-10-11 16:15:14.165014 | controller-0 | level=info msg=Cluster operator ingress Progressing is True with Reconciling: ingresscontroller "default" is progressing: IngressControllerProgressing: One or more status conditions indicate progressing: DeploymentRollingOut=True (DeploymentRollingOut: Waiting for router deployment rollout to finish: 0 of 2 updated replica(s) are available...
2025-10-11 16:15:14.165039 | controller-0 | level=info msg=).
2025-10-11 16:15:14.165063 | controller-0 | level=info msg=Not all ingress controllers are available.
2025-10-11 16:15:14.165115 | controller-0 | level=error msg=Cluster operator ingress Degraded is True with IngressDegraded: The "default" ingress controller reports Degraded=True: DegradedConditions: One or more other status conditions indicate a degraded state: DeploymentAvailable=False (DeploymentUnavailable: The deployment has Available status condition set to False (reason: MinimumReplicasUnavailable) with message: Deployment does not have minimum availability.), DeploymentReplicasMinAvailable=False (DeploymentMinimumReplicasNotMet: 0/2 of replicas are available, max unavailable is 1: Some pods are not scheduled: Pod "router-default-5ddb89f76-w6gvm" cannot be scheduled: 0/1 nodes are available: 1 node(s) didn't have free ports for the requested pod ports. preemption: 0/1 nodes are available: 1 node(s) didn't have free ports for the requested pod ports. Make sure you have sufficient worker nodes.)
2025-10-11 16:15:14.165180 | controller-0 | level=info msg=Cluster operator ingress EvaluationConditionsDetected is False with AsExpected:
2025-10-11 16:15:14.165245 | controller-0 | level=info msg=Cluster operator insights ClusterTransferAvailable is False with NoClusterTransfer: no available cluster transfer
2025-10-11 16:15:14.165293 | controller-0 | level=info msg=Cluster operator insights Disabled is False with AsExpected:
2025-10-11 16:15:14.165341 | controller-0 | level=info msg=Cluster operator insights RemoteConfigurationAvailable is True with AsExpected:
2025-10-11 16:15:14.165380 | controller-0 | level=info msg=Cluster operator insights RemoteConfigurationValid is True with AsExpected:
2025-10-11 16:15:14.165422 | controller-0 | level=info msg=Cluster operator insights SCAAvailable is True with Updated: SCA certs successfully updated in the etc-pki-entitlement secret
2025-10-11 16:15:14.165466 | controller-0 | level=info msg=Cluster operator kube-apiserver EvaluationConditionsDetected is False with AsExpected: All is well
2025-10-11 16:15:14.165504 | controller-0 | level=info msg=Cluster operator kube-controller-manager EvaluationConditionsDetected is Unknown with NoData:
2025-10-11 16:15:14.165546 | controller-0 | level=info msg=Cluster operator kube-scheduler EvaluationConditionsDetected is Unknown with NoData:
2025-10-11 16:15:14.165590 | controller-0 | level=info msg=Cluster operator kube-storage-version-migrator EvaluationConditionsDetected is Unknown with NoData:
2025-10-11 16:15:14.165639 | controller-0 | level=info msg=Cluster operator machine-config EvaluationConditionsDetected is False with AsExpected:
2025-10-11 16:15:14.165678 | controller-0 | level=info msg=Cluster operator monitoring Available is False with UpdatingPrometheusOperatorFailed: UpdatingPrometheusOperator: reconciling Prometheus Operator Admission Webhook Deployment failed: updating Deployment object failed: waiting for DeploymentRollout of openshift-monitoring/prometheus-operator-admission-webhook: context deadline exceeded: got 1 unavailable replicas
2025-10-11 16:15:14.166365 | controller-0 | level=error msg=Cluster operator monitoring Degraded is True with UpdatingPrometheusOperatorFailed: UpdatingPrometheusOperator: reconciling Prometheus Operator Admission Webhook Deployment failed: updating Deployment object failed: waiting for DeploymentRollout of openshift-monitoring/prometheus-operator-admission-webhook: context deadline exceeded: got 1 unavailable replicas
2025-10-11 16:15:14.166416 | controller-0 | level=info msg=Cluster operator monitoring Progressing is True with RollOutInProgress: Rolling out the stack.
2025-10-11 16:15:14.166455 | controller-0 | level=info msg=Cluster operator network ManagementStateDegraded is False with :
2025-10-11 16:15:14.166496 | controller-0 | level=info msg=Cluster operator network Progressing is True with Deploying: Deployment "/openshift-ovn-kubernetes/ovnkube-control-plane" is not available (awaiting 1 nodes)
2025-10-11 16:15:14.166539 | controller-0 | level=info msg=Cluster operator network Available is False with Startup: The network is starting up
2025-10-11 16:15:14.166585 | controller-0 | level=error msg=Cluster operator olm Degraded is True with CatalogdClusterCatalogOpenshiftCertifiedOperators_SyncError::CatalogdClusterCatalogOpenshiftCommunityOperators_SyncError::CatalogdClusterCatalogOpenshiftRedhatMarketplace_SyncError::CatalogdClusterCatalogOpenshiftRedhatOperators_SyncError: CatalogdClusterCatalogOpenshiftCertifiedOperatorsDegraded: Internal error occurred: failed calling webhook "inject-metadata-name.olm.operatorframework.io": failed to call webhook: Post "https://catalogd-service.openshift-catalogd.svc:9443/mutate-olm-operatorframework-io-v1-clustercatalog?timeout=10s": no endpoints available for service "catalogd-service"
2025-10-11 16:15:14.166618 | controller-0 | level=error msg=CatalogdClusterCatalogOpenshiftCommunityOperatorsDegraded: Internal error occurred: failed calling webhook "inject-metadata-name.olm.operatorframework.io": failed to call webhook: Post "https://catalogd-service.openshift-catalogd.svc:9443/mutate-olm-operatorframework-io-v1-clustercatalog?timeout=10s": no endpoints available for service "catalogd-service"
2025-10-11 16:15:14.166643 | controller-0 | level=error msg=CatalogdClusterCatalogOpenshiftRedhatMarketplaceDegraded: Internal error occurred: failed calling webhook "inject-metadata-name.olm.operatorframework.io": failed to call webhook: Post "https://catalogd-service.openshift-catalogd.svc:9443/mutate-olm-operatorframework-io-v1-clustercatalog?timeout=10s": no endpoints available for service "catalogd-service"
2025-10-11 16:15:14.166667 | controller-0 | level=error msg=CatalogdClusterCatalogOpenshiftRedhatOperatorsDegraded: Internal error occurred: failed calling webhook "inject-metadata-name.olm.operatorframework.io": failed to call webhook: Post "https://catalogd-service.openshift-catalogd.svc:9443/mutate-olm-operatorframework-io-v1-clustercatalog?timeout=10s": no endpoints available for service "catalogd-service"
2025-10-11 16:15:14.166746 | controller-0 | level=info msg=Cluster operator olm Progressing is True with CatalogdDeploymentCatalogdControllerManager_Deploying::OperatorcontrollerDeploymentOperatorControllerControllerManager_Deploying: CatalogdDeploymentCatalogdControllerManagerProgressing: Waiting for Deployment to deploy pods
2025-10-11 16:15:14.166771 | controller-0 | level=info msg=OperatorcontrollerDeploymentOperatorControllerControllerManagerProgressing: Waiting for Deployment to deploy pods
2025-10-11 16:15:14.166816 | controller-0 | level=info msg=Cluster operator olm Available is False with CatalogdDeploymentCatalogdControllerManager_Deploying::OperatorcontrollerDeploymentOperatorControllerControllerManager_Deploying: CatalogdDeploymentCatalogdControllerManagerAvailable: Waiting for Deployment
2025-10-11 16:15:14.166841 | controller-0 | level=info msg=OperatorcontrollerDeploymentOperatorControllerControllerManagerAvailable: Waiting for Deployment
2025-10-11 16:15:14.166896 | controller-0 | level=info msg=Cluster operator olm EvaluationConditionsDetected is Unknown with NoData:
2025-10-11 16:15:14.166936 | controller-0 | level=info msg=Cluster operator openshift-apiserver Available is False with APIServices_PreconditionNotReady: APIServicesAvailable: PreconditionNotReady
2025-10-11 16:15:14.166979 | controller-0 | level=info msg=Cluster operator openshift-apiserver EvaluationConditionsDetected is Unknown with NoData:
2025-10-11 16:15:14.167020 | controller-0 | level=info msg=Cluster operator openshift-controller-manager EvaluationConditionsDetected is Unknown with NoData:
2025-10-11 16:15:14.167061 | controller-0 | level=info msg=Cluster operator operator-lifecycle-manager-packageserver Available is False with ClusterServiceVersionNotSucceeded: ClusterServiceVersion openshift-operator-lifecycle-manager/packageserver observed in phase Failed with reason: InstallCheckFailed, message: install failed: deployment packageserver not ready before timeout: deployment "packageserver" exceeded its progress deadline
2025-10-11 16:15:14.167122 | controller-0 | level=info msg=Cluster operator service-ca EvaluationConditionsDetected is Unknown with NoData:
2025-10-11 16:15:14.167162 | controller-0 | level=info msg=Cluster operator storage EvaluationConditionsDetected is Unknown with NoData:
2025-10-11 16:15:14.167222 | controller-0 | level=info msg=Use the following commands to gather logs from the cluster
2025-10-11 16:15:14.167273 | controller-0 | level=info msg=openshift-install gather bootstrap --help
2025-10-11 16:15:14.167330 | controller-0 | level=error msg=Bootstrap failed to complete: : bootstrap process timed out: context deadline exceeded
2025-10-11 16:15:14.261847 | hotstack -> 38.102.83.64 | ERROR
2025-10-11 16:15:14.262103 | hotstack -> 38.102.83.64 | {
2025-10-11 16:15:14.262164 | hotstack -> 38.102.83.64 |   "delta": "1:00:00.194852",
2025-10-11 16:15:14.262206 | hotstack -> 38.102.83.64 |   "end": "2025-10-11 16:15:14.181937",
2025-10-11 16:15:14.262243 | hotstack -> 38.102.83.64 |   "msg": "non-zero return code",
2025-10-11 16:15:14.262283 | hotstack -> 38.102.83.64 |   "rc": 5,
2025-10-11 16:15:14.262318 | hotstack -> 38.102.83.64 |   "start": "2025-10-11 15:15:13.987085"
2025-10-11 16:15:14.262353 | hotstack -> 38.102.83.64 | }
failure
2025-10-11 16:15:14.266520 | 
2025-10-11 16:15:14.266615 | PLAY RECAP
2025-10-11 16:15:14.266726 | hotstack | ok: 74 changed: 48 unreachable: 0 failed: 1 skipped: 12 rescued: 0 ignored: 0
2025-10-11 16:15:14.266782 | 
2025-10-11 16:15:14.406019 | PRE-RUN END RESULT_NORMAL: [trusted : review.rdoproject.org/config/playbooks/hotstack/pre_stage2.yaml@master]
2025-10-11 16:15:14.412112 | POST-RUN START: [trusted : review.rdoproject.org/config/playbooks/hotstack/post.yaml@master]
2025-10-11 16:15:15.017407 | 
2025-10-11 16:15:15.017516 | PLAY [Collect hotstack logs and clean up environment]
2025-10-11 16:15:15.039134 | 
2025-10-11 16:15:15.039256 | TASK [Fetch stack outputs from remote to zuul executor]
2025-10-11 16:15:15.747777 | hotstack | Output suppressed because no_log was given
2025-10-11 16:15:15.759046 | 
2025-10-11 16:15:15.759200 | TASK [Load stack output vars with template resolution]
2025-10-11 16:15:15.822977 | hotstack | Output suppressed because no_log was given
2025-10-11 16:15:15.834107 | 
2025-10-11 16:15:15.834243 | TASK [Add Hotstack controller-0 to ansible inventory]
2025-10-11 16:15:15.869219 | hotstack | Output suppressed because no_log was given
2025-10-11 16:15:15.876877 | 
2025-10-11 16:15:15.876963 | TASK [Wait for Hotstack controller-0 to be ready]
2025-10-11 16:15:21.781628 | hotstack -> 38.102.83.64 | Output suppressed because no_log was given
2025-10-11 16:15:21.787342 | 
2025-10-11 16:15:21.787406 | TASK [Collect hotstack logs]
2025-10-11 16:15:21.810596 | hotstack | Output suppressed because no_log was given
2025-10-11 16:15:21.835957 | 
2025-10-11 16:15:21.836030 | TASK [hotlogs : Initialize failure tracking]
2025-10-11 16:15:21.864251 | hotstack | ok
2025-10-11 16:15:21.869160 | 
2025-10-11 16:15:21.869223 | TASK [hotlogs : Ensure logs directory]
2025-10-11 16:15:22.316557 | hotstack | changed
2025-10-11 16:15:22.321756 | 
2025-10-11 16:15:22.321824 | LOOP [hotlogs : Create hotlogs sub dirs]
2025-10-11 16:15:22.633564 | hotstack | changed:
2025-10-11 16:15:22.633858 | hotstack | {
2025-10-11 16:15:22.633896 | hotstack |   "dest_dir": "ocp_cluster",
2025-10-11 16:15:22.633922 | hotstack |   "src": "/home/zuul/ocp-cluster/.openshift_install.log"
2025-10-11 16:15:22.633947 | hotstack | }
2025-10-11 16:15:22.865553 | hotstack | ok:
2025-10-11 16:15:22.865668 | hotstack | {
2025-10-11 16:15:22.865870 | hotstack |   "dest_dir": "ocp_cluster",
2025-10-11 16:15:22.865926 | hotstack |   "src": "/home/zuul/cluster-custom-config/"
2025-10-11 16:15:22.865972 | hotstack | }
2025-10-11 16:15:23.119018 | hotstack | changed:
2025-10-11 16:15:23.119173 | hotstack | {
2025-10-11 16:15:23.119224 | hotstack |   "dest_dir": "data",
2025-10-11 16:15:23.119265 | hotstack |   "src": "/home/zuul/data/"
2025-10-11 16:15:23.119302 | hotstack | }
2025-10-11 16:15:23.396825 | hotstack | changed:
2025-10-11 16:15:23.397005 | hotstack | {
2025-10-11 16:15:23.397044 | hotstack |   "dest_dir": "manifests",
2025-10-11 16:15:23.397074 | hotstack |   "src": "/home/zuul/manifests/"
2025-10-11 16:15:23.397102 | hotstack | }
2025-10-11 16:15:23.657496 | hotstack | ok:
2025-10-11 16:15:23.657596 | hotstack | {
2025-10-11 16:15:23.657620 | hotstack |   "dest_dir": ".",
2025-10-11 16:15:23.657638 | hotstack |   "src": "/home/zuul/must-gather.tar.gz"
2025-10-11 16:15:23.657656 | hotstack | }
2025-10-11 16:15:23.667877 | 
2025-10-11 16:15:23.667941 | TASK [hotlogs : Run must-gather operations]
2025-10-11 16:15:24.507257 | hotstack -> 38.102.83.64 | ERROR
2025-10-11 16:15:24.507757 | hotstack -> 38.102.83.64 | {
2025-10-11 16:15:24.507831 | hotstack -> 38.102.83.64 |   "msg": "Must-gather failed: error: unable to resolve image stream 'openshift/must-gather': the server could not find the requested resource (get imagestreams.image.openshift.io must-gather)\n (Command: oc adm must-gather --image-stream=openshift/must-gather --image=quay.io/openstack-k8s-operators/openstack-must-gather --dest-dir=/home/zuul/must-gather --timeout=10m --host-network=false -- ADDITIONAL_NAMESPACES=sushy-emulator OPENSTACK_DATABASES=ALL SOS_EDPM=all SOS_DECOMPRESS=0 gather)",
2025-10-11 16:15:24.507881 | hotstack -> 38.102.83.64 |   "rc": 1
2025-10-11 16:15:24.507919 | hotstack -> 38.102.83.64 | }
2025-10-11 16:15:24.523075 | 
2025-10-11 16:15:24.523206 | TASK [hotlogs : Display must-gather failure]
2025-10-11 16:15:24.672222 | hotstack | ok:
2025-10-11 16:15:24.672375 | hotstack | Must-gather operations failed: Must-gather failed: error: unable to resolve image stream 'openshift/must-gather': the server could not find the requested resource (get imagestreams.image.openshift.io must-gather)
2025-10-11 16:15:24.672409 | hotstack |  (Command: oc adm must-gather --image-stream=openshift/must-gather --image=quay.io/openstack-k8s-operators/openstack-must-gather --dest-dir=/home/zuul/must-gather --timeout=10m --host-network=false -- ADDITIONAL_NAMESPACES=sushy-emulator OPENSTACK_DATABASES=ALL SOS_EDPM=all SOS_DECOMPRESS=0 gather)
2025-10-11 16:15:24.679029 | 
2025-10-11 16:15:24.679121 | TASK [hotlogs : Track must-gather failure]
2025-10-11 16:15:24.750892 | hotstack | ok
2025-10-11 16:15:24.758905 | 
2025-10-11 16:15:24.758991 | LOOP [hotlogs : Collect hotstack data]
2025-10-11 16:15:25.675328 | hotstack -> 38.102.83.70 | changed: >f+++++++++ .openshift_install.log
2025-10-11 16:15:25.675788 | hotstack -> 38.102.83.70 | changed: All items complete
2025-10-11 16:15:25.675848 | 
2025-10-11 16:15:26.290079 | hotstack -> 38.102.83.70 | changed:
2025-10-11 16:15:26.290246 | hotstack -> 38.102.83.70 | .d..t...... ./
2025-10-11 16:15:26.290287 | hotstack -> 38.102.83.70 | cd+++++++++ butane/
2025-10-11 16:15:26.290324 | hotstack -> 38.102.83.70 | >f+++++++++ butane/90-master-disable-netifnames.bu
2025-10-11 16:15:26.290357 | hotstack -> 38.102.83.70 | >f+++++++++ butane/90-master-enable-iscsi.bu
2025-10-11 16:15:26.290390 | hotstack -> 38.102.83.70 | >f+++++++++ butane/91-master-enable-multipath.bu
2025-10-11 16:15:26.290422 | hotstack -> 38.102.83.70 | >f+++++++++ butane/92-master-lv-cinder-volumes.bu
2025-10-11 16:15:26.290450 | hotstack -> 38.102.83.70 | cd+++++++++ config-assets/
2025-10-11 16:15:26.290480 | hotstack -> 38.102.83.70 | >f+++++++++ config-assets/95-etcd_config.yaml
2025-10-11 16:15:26.290512 | hotstack -> 38.102.83.70 | >f+++++++++ config-assets/ovn_k8s_config.yaml
2025-10-11 16:15:26.290541 | hotstack -> 38.102.83.70 | cd+++++++++ machine-configs/
2025-10-11 16:15:26.290571 | hotstack -> 38.102.83.70 | >f+++++++++ machine-configs/90-master-disable-netifnames.yaml
2025-10-11 16:15:26.290601 | hotstack -> 38.102.83.70 | >f+++++++++ machine-configs/90-master-enable-iscsi.yaml
2025-10-11 16:15:26.290630 | hotstack -> 38.102.83.70 | >f+++++++++ machine-configs/91-master-enable-multipath.yaml
2025-10-11 16:15:26.290659 | hotstack -> 38.102.83.70 | >f+++++++++ machine-configs/92-master-lv-cinder-volumes.yaml
2025-10-11 16:15:26.853318 | hotstack -> 38.102.83.70 | changed: .d..t...... ./
2025-10-11 16:15:27.532427 | hotstack -> 38.102.83.70 | ERROR: Item: {'src': '/home/zuul/manifests/', 'dest_dir': 'manifests'}
2025-10-11 16:15:27.532619 | hotstack -> 38.102.83.70 | {
2025-10-11 16:15:27.532647 | hotstack -> 38.102.83.70 |   "ansible_loop_var": "item",
2025-10-11 16:15:27.532665 | hotstack -> 38.102.83.70 |   "item": {
2025-10-11 16:15:27.532708 | hotstack -> 38.102.83.70 |     "dest_dir": "manifests",
2025-10-11 16:15:27.532726 | hotstack -> 38.102.83.70 |     "src": "/home/zuul/manifests/"
2025-10-11 16:15:27.532743 | hotstack -> 38.102.83.70 |   },
2025-10-11 16:15:27.532759 | hotstack -> 38.102.83.70 |   "msg": "Warning: Permanently added '38.102.83.64' (ED25519) to the list of known hosts.\r\nrsync: [sender] change_dir \"/home/zuul/manifests\" failed: No such file or directory (2)\nrsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1839) [Receiver=3.2.5]\nrsync: [Receiver] write error: Broken pipe (32)\n",
2025-10-11 16:15:27.532778 | hotstack -> 38.102.83.70 |   "rc": 23
2025-10-11 16:15:27.532794 | hotstack -> 38.102.83.70 | }
2025-10-11 16:15:28.223964 | hotstack -> 38.102.83.70 | ERROR: Item: {'src': '/home/zuul/must-gather.tar.gz', 'dest_dir': '.'}
2025-10-11 16:15:28.224404 | hotstack -> 38.102.83.70 | {
2025-10-11 16:15:28.224518 | hotstack -> 38.102.83.70 |   "ansible_loop_var": "item",
2025-10-11 16:15:28.224615 | hotstack -> 38.102.83.70 |   "item": {
2025-10-11 16:15:28.224732 | hotstack -> 38.102.83.70 |     "dest_dir": ".",
2025-10-11 16:15:28.224825 | hotstack -> 38.102.83.70 |     "src": "/home/zuul/must-gather.tar.gz"
2025-10-11 16:15:28.224914 | hotstack -> 38.102.83.70 |   },
2025-10-11 16:15:28.225003 | hotstack -> 38.102.83.70 |   "msg": "Warning: Permanently added '38.102.83.64' (ED25519) to the list of known hosts.\r\nrsync: [sender] link_stat \"/home/zuul/must-gather.tar.gz\" failed: No such file or directory (2)\nrsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1839) [Receiver=3.2.5]\nrsync: [Receiver] write error: Broken pipe (32)\n",
2025-10-11 16:15:28.225116 | hotstack -> 38.102.83.70 |   "rc": 23
2025-10-11 16:15:28.225208 | hotstack -> 38.102.83.70 | }
2025-10-11 16:15:28.238834 | hotstack | ERROR: Ignoring Errors
2025-10-11 16:15:28.253232 | 
2025-10-11 16:15:28.253383 | LOOP [hotlogs : Check hotstack data collection failures]
2025-10-11 16:15:28.322518 | hotstack | skipping: Conditional result was False
2025-10-11 16:15:28.334184 | hotstack | skipping: Conditional result was False
2025-10-11 16:15:28.344770 | hotstack | skipping: Conditional result was False
2025-10-11 16:15:28.373281 | hotstack | ERROR
2025-10-11 16:15:28.373508 | hotstack | {
2025-10-11 16:15:28.373565 | hotstack |   "msg": "Unexpected templating type error occurred on ({{ hotlogs_failures + ['Failed to collect: ' + item.item] }}): can only concatenate str (not \"dict\") to str. can only concatenate str (not \"dict\") to str"
2025-10-11 16:15:28.373607 | hotstack | }
2025-10-11 16:15:28.383110 | 
2025-10-11 16:15:28.383190 | TASK [Print message that log collection failed]
2025-10-11 16:15:28.414086 | hotstack | ok: WARNING: Log collection failed and may be incomplete. Continue with cleanup ...
2025-10-11 16:15:28.420418 | 
2025-10-11 16:15:28.420496 | TASK [Set _auth_type fact]
2025-10-11 16:15:28.479558 | hotstack | Output suppressed because no_log was given
2025-10-11 16:15:28.486712 | 
2025-10-11 16:15:28.486793 | TASK [Write clouds.yaml]
2025-10-11 16:15:29.650125 | hotstack | Output suppressed because no_log was given
2025-10-11 16:15:29.661128 | 
2025-10-11 16:15:29.661255 | TASK [Delete application credentials]
2025-10-11 16:15:31.708433 | hotstack | changed
2025-10-11 16:15:31.719615 | 
2025-10-11 16:15:31.719770 | TASK [Print message that application credential failed to delete]
2025-10-11 16:15:31.746478 | hotstack | skipping: Conditional result was False
2025-10-11 16:15:31.758762 | 
2025-10-11 16:15:31.758889 | TASK [Print message that application credential deleted successfully]
2025-10-11 16:15:31.806871 | hotstack | ok: Application credential deleted successfully.
2025-10-11 16:15:31.817162 | 
2025-10-11 16:15:31.817289 | TASK [Write clouds.yaml with application credentials]
2025-10-11 16:15:31.856098 | 
2025-10-11 16:15:31.856406 | TASK [Delete the heat stack]
2025-10-11 12:15:59.561537 | hotstack | 2025-10-11 16:15:34Z [hs-918b704357334bf18129d9b4a9d53906]: DELETE_IN_PROGRESS  Stack DELETE started
2025-10-11 12:15:59.561734 | hotstack | 2025-10-11 16:15:35Z [hs-918b704357334bf18129d9b4a9d53906.networker1]: DELETE_IN_PROGRESS  state changed
2025-10-11 12:15:59.561753 | hotstack | 2025-10-11 16:15:36Z [hs-918b704357334bf18129d9b4a9d53906.networker2]: DELETE_IN_PROGRESS  state changed
2025-10-11 12:15:59.561767 | hotstack | 2025-10-11 16:15:37Z [hs-918b704357334bf18129d9b4a9d53906.master1]: DELETE_IN_PROGRESS  state changed
2025-10-11 12:15:59.561781 | hotstack | 2025-10-11 16:15:38Z [hs-918b704357334bf18129d9b4a9d53906.controller]: DELETE_IN_PROGRESS  state changed
2025-10-11 12:15:59.561795 | hotstack | 2025-10-11 16:15:39Z [hs-918b704357334bf18129d9b4a9d53906.compute0]: DELETE_IN_PROGRESS  state changed
2025-10-11 12:15:59.561808 | hotstack | 2025-10-11 16:15:40Z [hs-918b704357334bf18129d9b4a9d53906.master0]: DELETE_IN_PROGRESS  state changed
2025-10-11 12:15:59.561821 | hotstack | 2025-10-11 16:15:41Z [hs-918b704357334bf18129d9b4a9d53906.networker0]: DELETE_IN_PROGRESS  state changed
2025-10-11 12:15:59.561834 | hotstack | 2025-10-11 16:15:42Z [hs-918b704357334bf18129d9b4a9d53906.compute1]: DELETE_IN_PROGRESS  state changed
2025-10-11 12:15:59.561847 | hotstack | 2025-10-11 16:15:42Z [hs-918b704357334bf18129d9b4a9d53906.networker2]: DELETE_COMPLETE  state changed
2025-10-11 12:15:59.561861 | hotstack | 2025-10-11 16:15:42Z [hs-918b704357334bf18129d9b4a9d53906.networker2-trunk0]: DELETE_IN_PROGRESS  state changed
2025-10-11 12:15:59.561874 | hotstack | 2025-10-11 16:15:43Z [hs-918b704357334bf18129d9b4a9d53906.master2]: DELETE_IN_PROGRESS  state changed
2025-10-11 12:15:59.561887 | hotstack | 2025-10-11 16:15:43Z [hs-918b704357334bf18129d9b4a9d53906.networker2-trunk0]: DELETE_COMPLETE  state changed
2025-10-11 12:15:59.561901 | hotstack | 2025-10-11 16:15:43Z [hs-918b704357334bf18129d9b4a9d53906.networker2-internal-api-port]: DELETE_IN_PROGRESS  state changed
2025-10-11 12:15:59.561915 | hotstack | 2025-10-11 16:15:43Z [hs-918b704357334bf18129d9b4a9d53906.networker2-ctlplane-trunk-parent-port]: DELETE_IN_PROGRESS  state changed
2025-10-11 12:15:59.561929 | hotstack | 2025-10-11 16:15:43Z [hs-918b704357334bf18129d9b4a9d53906.networker2-tenant-port]: DELETE_IN_PROGRESS  state changed
2025-10-11 12:15:59.561942 | hotstack | 2025-10-11 16:15:43Z [hs-918b704357334bf18129d9b4a9d53906.networker1]: DELETE_COMPLETE  state changed
2025-10-11 12:15:59.561956 | hotstack | 2025-10-11 16:15:44Z [hs-918b704357334bf18129d9b4a9d53906.networker1-trunk0]: DELETE_IN_PROGRESS  state changed
2025-10-11 12:15:59.561969 | hotstack | 2025-10-11 16:15:44Z [hs-918b704357334bf18129d9b4a9d53906.ctlplane-net-router-interface]: DELETE_IN_PROGRESS  state changed
2025-10-11 12:15:59.561982 | hotstack | 2025-10-11 16:15:44Z [hs-918b704357334bf18129d9b4a9d53906.networker2-tenant-port]: DELETE_COMPLETE  state changed
2025-10-11 12:15:59.561995 | hotstack | 2025-10-11 16:15:44Z [hs-918b704357334bf18129d9b4a9d53906.networker2-internal-api-port]: DELETE_COMPLETE  state changed
2025-10-11 12:15:59.562008 | hotstack | 2025-10-11 16:15:44Z [hs-918b704357334bf18129d9b4a9d53906.networker2-ctlplane-trunk-parent-port]: DELETE_COMPLETE  state changed
2025-10-11 12:15:59.562021 | hotstack | 2025-10-11 16:15:45Z [hs-918b704357334bf18129d9b4a9d53906.controller-floating-ip]: DELETE_IN_PROGRESS  state changed
2025-10-11 12:15:59.562034 | hotstack | 2025-10-11 16:15:45Z [hs-918b704357334bf18129d9b4a9d53906.controller]: DELETE_COMPLETE  state changed
2025-10-11 12:15:59.562047 | hotstack | 2025-10-11 16:15:45Z [hs-918b704357334bf18129d9b4a9d53906.controller-init]: DELETE_IN_PROGRESS  state changed
2025-10-11 12:15:59.562060 | hotstack | 2025-10-11 16:15:45Z [hs-918b704357334bf18129d9b4a9d53906.controller-init]: DELETE_COMPLETE  state changed
2025-10-11 12:15:59.562074 | hotstack | 2025-10-11 16:15:45Z [hs-918b704357334bf18129d9b4a9d53906.controller-write-files]: DELETE_IN_PROGRESS  state changed
2025-10-11 12:15:59.562087 | hotstack | 2025-10-11 16:15:45Z [hs-918b704357334bf18129d9b4a9d53906.controller_users]: DELETE_IN_PROGRESS  state changed
2025-10-11 12:15:59.562100 | hotstack | 2025-10-11 16:15:45Z [hs-918b704357334bf18129d9b4a9d53906.controller-runcmd]: DELETE_IN_PROGRESS  state changed
2025-10-11 12:15:59.562113 | hotstack | 2025-10-11 16:15:45Z [hs-918b704357334bf18129d9b4a9d53906.controller-write-files]: DELETE_COMPLETE  state changed
2025-10-11 12:15:59.562144 | hotstack | 2025-10-11 16:15:45Z [hs-918b704357334bf18129d9b4a9d53906.controller-runcmd]: DELETE_COMPLETE  state changed
2025-10-11 12:15:59.562158 | hotstack | 2025-10-11 16:15:45Z [hs-918b704357334bf18129d9b4a9d53906.controller_users]: DELETE_COMPLETE  state changed
2025-10-11 12:15:59.562234 | hotstack | 2025-10-11 16:15:46Z [hs-918b704357334bf18129d9b4a9d53906.ironic-net-router-interface]: DELETE_IN_PROGRESS  state changed
2025-10-11 12:15:59.562256 | hotstack | 2025-10-11 16:15:46Z [hs-918b704357334bf18129d9b4a9d53906.networker1-trunk0]: DELETE_COMPLETE  state changed
2025-10-11 12:15:59.562270 | hotstack | 2025-10-11 16:15:46Z [hs-918b704357334bf18129d9b4a9d53906.networker1-tenant-port]: DELETE_IN_PROGRESS  state changed
2025-10-11 12:15:59.562283 | hotstack | 2025-10-11 16:15:46Z [hs-918b704357334bf18129d9b4a9d53906.networker1-internal-api-port]: DELETE_IN_PROGRESS  state changed
2025-10-11 12:15:59.562345 | hotstack | 2025-10-11 16:15:46Z [hs-918b704357334bf18129d9b4a9d53906.networker1-ctlplane-trunk-parent-port]: DELETE_IN_PROGRESS  state changed
2025-10-11 12:16:14.907726 | hotstack | 2025-10-11 16:15:46Z [hs-918b704357334bf18129d9b4a9d53906.controller-floating-ip]: DELETE_COMPLETE  state changed
2025-10-11 12:16:14.907864 | hotstack | 2025-10-11 16:15:47Z [hs-918b704357334bf18129d9b4a9d53906.machine-net-router-interface]: DELETE_IN_PROGRESS  state changed
2025-10-11 12:16:14.907875 | hotstack | 2025-10-11 16:15:47Z [hs-918b704357334bf18129d9b4a9d53906.ctlplane-net-router-interface]: DELETE_COMPLETE  state changed
2025-10-11 12:16:14.907885 | hotstack | 2025-10-11 16:15:47Z [hs-918b704357334bf18129d9b4a9d53906.networker1-tenant-port]: DELETE_COMPLETE  state changed
2025-10-11 12:16:14.907893 | hotstack | 2025-10-11 16:15:49Z [hs-918b704357334bf18129d9b4a9d53906.networker1-internal-api-port]: DELETE_COMPLETE  state changed
2025-10-11 12:16:14.907914 | hotstack | 2025-10-11 16:15:50Z [hs-918b704357334bf18129d9b4a9d53906.networker0]: DELETE_COMPLETE  state changed
2025-10-11 12:16:14.907922 | hotstack | 2025-10-11 16:15:50Z [hs-918b704357334bf18129d9b4a9d53906.networker0-trunk0]: DELETE_IN_PROGRESS  state changed
2025-10-11 12:16:14.907930 | hotstack | 2025-10-11 16:15:50Z [hs-918b704357334bf18129d9b4a9d53906.networker_init]: DELETE_IN_PROGRESS  state changed
2025-10-11 12:16:14.907953 | hotstack | 2025-10-11 16:15:50Z [hs-918b704357334bf18129d9b4a9d53906.networker_init]: DELETE_COMPLETE  state changed
2025-10-11 12:16:14.907961 | hotstack | 2025-10-11 16:15:51Z [hs-918b704357334bf18129d9b4a9d53906.networker0-trunk0]: DELETE_COMPLETE  state changed
2025-10-11 12:16:14.907969 | hotstack | 2025-10-11 16:15:51Z [hs-918b704357334bf18129d9b4a9d53906.networker0-tenant-port]: DELETE_IN_PROGRESS  state changed
2025-10-11 12:16:14.907978 | hotstack | 2025-10-11 16:15:51Z [hs-918b704357334bf18129d9b4a9d53906.networker0-internal-api-port]: DELETE_IN_PROGRESS  state changed
2025-10-11 12:16:14.907985 | hotstack | 2025-10-11 16:15:51Z [hs-918b704357334bf18129d9b4a9d53906.compute1]: DELETE_COMPLETE  state changed
2025-10-11 12:16:14.907994 | hotstack | 2025-10-11 16:15:51Z [hs-918b704357334bf18129d9b4a9d53906.networker0-ctlplane-trunk-parent-port]: DELETE_IN_PROGRESS  state changed
2025-10-11 12:16:14.908003 | hotstack | 2025-10-11 16:15:51Z [hs-918b704357334bf18129d9b4a9d53906.compute1_init]: DELETE_IN_PROGRESS  state changed
2025-10-11 12:16:14.908012 | hotstack | 2025-10-11 16:15:51Z [hs-918b704357334bf18129d9b4a9d53906.compute1-trunk0]: DELETE_IN_PROGRESS  state changed
2025-10-11 12:16:14.908020 | hotstack | 2025-10-11 16:15:51Z [hs-918b704357334bf18129d9b4a9d53906.compute1_init]: DELETE_COMPLETE  state changed
2025-10-11 12:16:14.908029 | hotstack | 2025-10-11 16:15:51Z [hs-918b704357334bf18129d9b4a9d53906.compute1_hostname]: DELETE_IN_PROGRESS  state changed
2025-10-11 12:16:14.908037 | hotstack | 2025-10-11 16:15:51Z [hs-918b704357334bf18129d9b4a9d53906.compute1_hostname]: DELETE_COMPLETE  state changed
2025-10-11 12:16:14.908045 | hotstack | 2025-10-11 16:15:52Z [hs-918b704357334bf18129d9b4a9d53906.networker0-internal-api-port]: DELETE_COMPLETE  state changed
2025-10-11 12:16:14.908052 | hotstack | 2025-10-11 16:15:52Z [hs-918b704357334bf18129d9b4a9d53906.ironic-net-router-interface]: DELETE_COMPLETE  state changed
2025-10-11 12:16:14.908079 | hotstack | 2025-10-11 16:15:52Z [hs-918b704357334bf18129d9b4a9d53906.networker0-tenant-port]: DELETE_COMPLETE  state changed
2025-10-11 12:16:14.908088 | hotstack | 2025-10-11 16:15:53Z [hs-918b704357334bf18129d9b4a9d53906.compute1-trunk0]: DELETE_COMPLETE  state changed
2025-10-11 12:16:14.908095 | hotstack | 2025-10-11 16:15:53Z [hs-918b704357334bf18129d9b4a9d53906.compute1-storage-port]: DELETE_IN_PROGRESS  state changed
2025-10-11 12:16:14.908103 | hotstack | 2025-10-11 16:15:53Z [hs-918b704357334bf18129d9b4a9d53906.networker0-ctlplane-trunk-parent-port]: DELETE_COMPLETE  state changed
2025-10-11 12:16:14.908112 | hotstack | 2025-10-11 16:15:53Z [hs-918b704357334bf18129d9b4a9d53906.compute1-internal-api-port]: DELETE_IN_PROGRESS  state changed
2025-10-11 12:16:14.908120 | hotstack | 2025-10-11 16:15:53Z [hs-918b704357334bf18129d9b4a9d53906.machine-net-router-interface]: DELETE_COMPLETE  state changed
2025-10-11 12:16:14.908162 | hotstack | 2025-10-11 16:15:53Z [hs-918b704357334bf18129d9b4a9d53906.compute1-ctlplane-trunk-parent-port]: DELETE_IN_PROGRESS  state changed
2025-10-11 12:16:14.908225 | hotstack | 2025-10-11 16:15:53Z [hs-918b704357334bf18129d9b4a9d53906.compute1-tenant-port]: DELETE_IN_PROGRESS  state changed
2025-10-11 12:16:14.908241 | hotstack | 2025-10-11 16:15:53Z [hs-918b704357334bf18129d9b4a9d53906.router]: DELETE_IN_PROGRESS  state changed
2025-10-11 12:16:14.908249 | hotstack | 2025-10-11 16:15:54Z [hs-918b704357334bf18129d9b4a9d53906.networker1-ctlplane-trunk-parent-port]: DELETE_COMPLETE  state changed
2025-10-11 12:16:14.908257 | hotstack | 2025-10-11 16:15:54Z [hs-918b704357334bf18129d9b4a9d53906.compute1-internal-api-port]: DELETE_COMPLETE  state changed
2025-10-11 12:16:14.908264 | hotstack | 2025-10-11 16:15:54Z [hs-918b704357334bf18129d9b4a9d53906.compute1-storage-port]: DELETE_COMPLETE  state changed
2025-10-11 12:16:14.908272 | hotstack | 2025-10-11 16:15:54Z [hs-918b704357334bf18129d9b4a9d53906.compute1-tenant-port]: DELETE_COMPLETE  state changed
2025-10-11 12:16:14.908280 | hotstack | 2025-10-11 16:15:54Z [hs-918b704357334bf18129d9b4a9d53906.compute1-ctlplane-trunk-parent-port]: DELETE_COMPLETE  state changed
2025-10-11 12:16:14.908311 | hotstack | 2025-10-11 16:15:55Z [hs-918b704357334bf18129d9b4a9d53906.router]: DELETE_COMPLETE  state changed
2025-10-11 12:16:14.908319 | hotstack | 2025-10-11 16:15:56Z [hs-918b704357334bf18129d9b4a9d53906.compute0]: DELETE_COMPLETE  state changed
2025-10-11 12:16:14.908326 | hotstack | 2025-10-11 16:15:56Z [hs-918b704357334bf18129d9b4a9d53906.compute0-trunk0]: DELETE_IN_PROGRESS  state changed
2025-10-11 12:16:14.908334 | hotstack | 2025-10-11 16:15:56Z [hs-918b704357334bf18129d9b4a9d53906.compute0_init]: DELETE_IN_PROGRESS  state changed
2025-10-11 12:16:14.908343 | hotstack | 2025-10-11 16:15:56Z [hs-918b704357334bf18129d9b4a9d53906.compute0_init]: DELETE_COMPLETE  state changed
2025-10-11 12:16:14.908351 | hotstack | 2025-10-11 16:15:56Z [hs-918b704357334bf18129d9b4a9d53906.compute0_hostname]: DELETE_IN_PROGRESS  state changed
2025-10-11 12:16:14.908358 | hotstack | 2025-10-11 16:15:56Z [hs-918b704357334bf18129d9b4a9d53906.dataplane_users]: DELETE_IN_PROGRESS  state changed
2025-10-11 12:16:14.908366 | hotstack | 2025-10-11 16:15:56Z [hs-918b704357334bf18129d9b4a9d53906.compute0_hostname]: DELETE_COMPLETE  state changed
2025-10-11 12:16:14.908374 | hotstack | 2025-10-11 16:15:56Z [hs-918b704357334bf18129d9b4a9d53906.dataplane_users]: DELETE_COMPLETE  state changed
2025-10-11 12:16:14.908382 | hotstack | 2025-10-11 16:15:57Z [hs-918b704357334bf18129d9b4a9d53906.compute0-trunk0]: DELETE_COMPLETE  state changed
2025-10-11 12:16:14.908389 | hotstack | 2025-10-11 16:15:57Z [hs-918b704357334bf18129d9b4a9d53906.compute0-ctlplane-trunk-parent-port]: DELETE_IN_PROGRESS  state changed
2025-10-11 12:16:14.908397 | hotstack | 2025-10-11 16:15:57Z [hs-918b704357334bf18129d9b4a9d53906.compute0-storage-port]: DELETE_IN_PROGRESS  state changed
2025-10-11 12:16:14.908408 | hotstack | 2025-10-11 16:15:57Z [hs-918b704357334bf18129d9b4a9d53906.compute0-internal-api-port]: DELETE_IN_PROGRESS  state changed
2025-10-11 12:16:14.908416 | hotstack | 2025-10-11 16:15:57Z [hs-918b704357334bf18129d9b4a9d53906.compute0-tenant-port]: DELETE_IN_PROGRESS  state changed
2025-10-11 12:16:14.908423 | hotstack | 2025-10-11 16:15:58Z [hs-918b704357334bf18129d9b4a9d53906.compute0-ctlplane-trunk-parent-port]: DELETE_COMPLETE  state changed
2025-10-11 12:16:14.908437 | hotstack | 2025-10-11 16:15:58Z [hs-918b704357334bf18129d9b4a9d53906.compute0-internal-api-port]: DELETE_COMPLETE  state changed
2025-10-11 12:16:14.908445 | hotstack | 2025-10-11 16:15:58Z [hs-918b704357334bf18129d9b4a9d53906.compute0-tenant-port]: DELETE_COMPLETE  state changed
2025-10-11 12:16:14.908453 | hotstack | 2025-10-11 16:15:58Z [hs-918b704357334bf18129d9b4a9d53906.compute0-storage-port]: DELETE_COMPLETE  state changed
2025-10-11 12:16:14.908461 | hotstack | 2025-10-11 16:15:59Z [hs-918b704357334bf18129d9b4a9d53906.master1]: DELETE_COMPLETE  state changed
2025-10-11 12:16:14.908469 | hotstack | 2025-10-11 16:15:59Z [hs-918b704357334bf18129d9b4a9d53906.master1-cinder-vol2]: DELETE_IN_PROGRESS  state changed
2025-10-11 12:16:14.908477 | hotstack | 2025-10-11 16:15:59Z [hs-918b704357334bf18129d9b4a9d53906.master1-cinder-vol0]: DELETE_IN_PROGRESS  state changed
2025-10-11 12:16:14.908485 | hotstack | 2025-10-11 16:15:59Z [hs-918b704357334bf18129d9b4a9d53906.master1-lvms-vol0]: DELETE_IN_PROGRESS  state changed
2025-10-11 12:16:14.908493 | hotstack | 2025-10-11 16:15:59Z [hs-918b704357334bf18129d9b4a9d53906.master1-machine-port]: DELETE_IN_PROGRESS  state changed
2025-10-11 12:16:14.908501 | hotstack | 2025-10-11 16:15:59Z [hs-918b704357334bf18129d9b4a9d53906.master1-cinder-vol1]: DELETE_IN_PROGRESS  state changed
2025-10-11 12:16:14.908508 | hotstack | 2025-10-11 16:15:59Z [hs-918b704357334bf18129d9b4a9d53906.master1-trunk0]: DELETE_IN_PROGRESS  state changed
2025-10-11 12:16:14.908516 | hotstack | 2025-10-11 16:15:59Z [hs-918b704357334bf18129d9b4a9d53906.master1-ironic-port]: DELETE_IN_PROGRESS  state changed
2025-10-11 12:16:14.908523 | hotstack | 2025-10-11 16:16:00Z [hs-918b704357334bf18129d9b4a9d53906.master1-machine-port]: DELETE_COMPLETE  state changed
2025-10-11 12:16:14.908531 | hotstack | 2025-10-11 16:16:00Z [hs-918b704357334bf18129d9b4a9d53906.master1-trunk0]: DELETE_COMPLETE  state changed
2025-10-11 12:16:14.908538 | hotstack | 2025-10-11 16:16:00Z [hs-918b704357334bf18129d9b4a9d53906.master1-internal-api-port]: DELETE_IN_PROGRESS  state changed
2025-10-11 12:16:14.908546 | hotstack | 2025-10-11 16:16:00Z [hs-918b704357334bf18129d9b4a9d53906.master1-octavia-port]: DELETE_IN_PROGRESS  state changed
2025-10-11 12:16:14.908554 | hotstack | 2025-10-11 16:16:01Z [hs-918b704357334bf18129d9b4a9d53906.master1-storage-port]: DELETE_IN_PROGRESS  state changed
2025-10-11 12:16:14.908562 | hotstack | 2025-10-11 16:16:01Z [hs-918b704357334bf18129d9b4a9d53906.master1-ctlplane-trunk-parent-port]: DELETE_IN_PROGRESS  state changed
2025-10-11 12:16:14.908570 | hotstack | 2025-10-11 16:16:01Z [hs-918b704357334bf18129d9b4a9d53906.master1-tenant-port]: DELETE_IN_PROGRESS  state changed
2025-10-11 12:16:14.908578 | hotstack | 2025-10-11 16:16:01Z [hs-918b704357334bf18129d9b4a9d53906.master1-ironic-port]: DELETE_COMPLETE  state changed
2025-10-11 12:16:14.908586 | hotstack | 2025-10-11 16:16:01Z [hs-918b704357334bf18129d9b4a9d53906.master1-internal-api-port]: DELETE_COMPLETE  state changed
2025-10-11 12:16:14.908594 | hotstack | 2025-10-11 16:16:01Z [hs-918b704357334bf18129d9b4a9d53906.master1-octavia-port]: DELETE_COMPLETE  state changed
2025-10-11 12:16:14.908612 | hotstack | 2025-10-11 16:16:01Z [hs-918b704357334bf18129d9b4a9d53906.master1-tenant-port]: DELETE_COMPLETE  state changed
2025-10-11 12:16:20.021225 | hotstack | 2025-10-11 16:16:01Z [hs-918b704357334bf18129d9b4a9d53906.master1-storage-port]: DELETE_COMPLETE  state changed
2025-10-11 12:16:20.021364 | hotstack | 2025-10-11 16:16:02Z [hs-918b704357334bf18129d9b4a9d53906.master1-ctlplane-trunk-parent-port]: DELETE_COMPLETE  state changed
2025-10-11 12:16:20.021381 | hotstack | 2025-10-11 16:16:06Z [hs-918b704357334bf18129d9b4a9d53906.master0]: DELETE_COMPLETE  state changed
2025-10-11 12:16:20.021396 | hotstack | 2025-10-11 16:16:06Z [hs-918b704357334bf18129d9b4a9d53906.master0-cinder-vol0]: DELETE_IN_PROGRESS  state changed
2025-10-11 12:16:20.021437 | hotstack | 2025-10-11 16:16:06Z [hs-918b704357334bf18129d9b4a9d53906.master0-machine-port]: DELETE_IN_PROGRESS  state changed
2025-10-11 12:16:20.021451 | hotstack | 2025-10-11 16:16:06Z [hs-918b704357334bf18129d9b4a9d53906.master0-cinder-vol1]: DELETE_IN_PROGRESS  state changed
2025-10-11 12:16:20.021464 | hotstack | 2025-10-11 16:16:06Z [hs-918b704357334bf18129d9b4a9d53906.master0-ironic-port]: DELETE_IN_PROGRESS  state changed
2025-10-11 12:16:20.021496 | hotstack | 2025-10-11 16:16:06Z [hs-918b704357334bf18129d9b4a9d53906.master0-trunk0]: DELETE_IN_PROGRESS  state changed
2025-10-11 12:16:20.021510 | hotstack | 2025-10-11 16:16:06Z [hs-918b704357334bf18129d9b4a9d53906.master0-cinder-vol2]: DELETE_IN_PROGRESS  state changed
2025-10-11 12:16:20.021523 | hotstack | 2025-10-11 16:16:07Z [hs-918b704357334bf18129d9b4a9d53906.master0-lvms-vol0]: DELETE_IN_PROGRESS  state changed
2025-10-11 12:16:20.021536 | hotstack | 2025-10-11 16:16:07Z [hs-918b704357334bf18129d9b4a9d53906.master0-ironic-port]: DELETE_COMPLETE  state changed
2025-10-11 12:16:20.021550 | hotstack | 2025-10-11 16:16:07Z [hs-918b704357334bf18129d9b4a9d53906.master0-cinder-vol0]: DELETE_COMPLETE  state changed
2025-10-11 12:16:20.021563 | hotstack | 2025-10-11 16:16:07Z [hs-918b704357334bf18129d9b4a9d53906.master0-machine-port]: DELETE_COMPLETE  state changed
2025-10-11 12:16:20.021576 | hotstack | 2025-10-11 16:16:07Z [hs-918b704357334bf18129d9b4a9d53906.master0-cinder-vol1]: DELETE_COMPLETE  state changed
2025-10-11 12:16:20.021595 | hotstack | 2025-10-11 16:16:08Z [hs-918b704357334bf18129d9b4a9d53906.master0-cinder-vol2]: DELETE_COMPLETE  state changed
2025-10-11 12:16:20.021608 | hotstack | 2025-10-11 16:16:08Z [hs-918b704357334bf18129d9b4a9d53906.master0-lvms-vol0]: DELETE_COMPLETE  state changed
2025-10-11 12:16:20.021621 | hotstack | 2025-10-11 16:16:08Z [hs-918b704357334bf18129d9b4a9d53906.master0-trunk0]: DELETE_COMPLETE  state changed
2025-10-11 12:16:20.021635 | hotstack | 2025-10-11 16:16:08Z [hs-918b704357334bf18129d9b4a9d53906.master0-internal-api-port]: DELETE_IN_PROGRESS  state changed
2025-10-11 12:16:20.021649 | hotstack | 2025-10-11 16:16:08Z [hs-918b704357334bf18129d9b4a9d53906.master0-octavia-port]: DELETE_IN_PROGRESS  state changed
2025-10-11 12:16:20.021665 | hotstack | 2025-10-11 16:16:08Z [hs-918b704357334bf18129d9b4a9d53906.master0-storage-port]: DELETE_IN_PROGRESS  state changed
2025-10-11 12:16:20.021678 | hotstack | 2025-10-11 16:16:08Z [hs-918b704357334bf18129d9b4a9d53906.master0-tenant-port]: DELETE_IN_PROGRESS  state changed
2025-10-11 12:16:20.021691 | hotstack | 2025-10-11 16:16:08Z [hs-918b704357334bf18129d9b4a9d53906.master0-ctlplane-trunk-parent-port]: DELETE_IN_PROGRESS  state changed
2025-10-11 12:16:20.021705 | hotstack | 2025-10-11 16:16:09Z [hs-918b704357334bf18129d9b4a9d53906.master0-internal-api-port]: DELETE_COMPLETE  state changed
2025-10-11 12:16:20.021719 | hotstack | 2025-10-11 16:16:09Z [hs-918b704357334bf18129d9b4a9d53906.master0-octavia-port]: DELETE_COMPLETE  state changed
2025-10-11 12:16:20.021732 | hotstack | 2025-10-11 16:16:09Z [hs-918b704357334bf18129d9b4a9d53906.master0-storage-port]: DELETE_COMPLETE  state changed
2025-10-11 12:16:20.021745 | hotstack | 2025-10-11 16:16:09Z [hs-918b704357334bf18129d9b4a9d53906.master2]: DELETE_COMPLETE  state changed
2025-10-11 12:16:20.021758 | hotstack | 2025-10-11 16:16:09Z [hs-918b704357334bf18129d9b4a9d53906.master2-lvms-vol0]: DELETE_IN_PROGRESS  state changed
2025-10-11 12:16:20.021771 | hotstack | 2025-10-11 16:16:09Z [hs-918b704357334bf18129d9b4a9d53906.master2-machine-port]: DELETE_IN_PROGRESS  state changed
2025-10-11 12:16:20.021784 | hotstack | 2025-10-11 16:16:09Z [hs-918b704357334bf18129d9b4a9d53906.master2-cinder-vol1]: DELETE_IN_PROGRESS  state changed
2025-10-11 12:16:20.021798 | hotstack | 2025-10-11 16:16:09Z [hs-918b704357334bf18129d9b4a9d53906.master2-ironic-port]: DELETE_IN_PROGRESS  state changed
2025-10-11 12:16:20.021811 | hotstack | 2025-10-11 16:16:09Z [hs-918b704357334bf18129d9b4a9d53906.master2-cinder-vol0]: DELETE_IN_PROGRESS  state changed
2025-10-11 12:16:20.021824 | hotstack | 2025-10-11 16:16:09Z [hs-918b704357334bf18129d9b4a9d53906.master2-trunk0]: DELETE_IN_PROGRESS  state changed
2025-10-11 12:16:20.021837 | hotstack | 2025-10-11 16:16:09Z [hs-918b704357334bf18129d9b4a9d53906.master2-cinder-vol2]: DELETE_IN_PROGRESS  state changed
2025-10-11 12:16:20.021850 | hotstack | 2025-10-11 16:16:10Z [hs-918b704357334bf18129d9b4a9d53906.master0-tenant-port]: DELETE_COMPLETE  state changed
2025-10-11 12:16:20.021887 | hotstack | 2025-10-11 16:16:10Z [hs-918b704357334bf18129d9b4a9d53906.master0-ctlplane-trunk-parent-port]: DELETE_COMPLETE  state changed
2025-10-11 12:16:20.021908 | hotstack | 2025-10-11 16:16:10Z [hs-918b704357334bf18129d9b4a9d53906.master2-ironic-port]: DELETE_COMPLETE  state changed
2025-10-11 12:16:20.021922 | hotstack | 2025-10-11 16:16:10Z [hs-918b704357334bf18129d9b4a9d53906.ironic-subnet]: DELETE_IN_PROGRESS  state changed
2025-10-11 12:16:20.021935 | hotstack | 2025-10-11 16:16:10Z [hs-918b704357334bf18129d9b4a9d53906.master2-machine-port]: DELETE_COMPLETE  state changed
2025-10-11 12:16:20.021948 | hotstack | 2025-10-11 16:16:10Z [hs-918b704357334bf18129d9b4a9d53906.extra-dhcp-opts-value]: DELETE_IN_PROGRESS  state changed
2025-10-11 12:16:20.021961 | hotstack | 2025-10-11 16:16:10Z [hs-918b704357334bf18129d9b4a9d53906.master2-lvms-vol0]: DELETE_COMPLETE  state changed
2025-10-11 12:16:20.021975 | hotstack | 2025-10-11 16:16:10Z [hs-918b704357334bf18129d9b4a9d53906.extra-dhcp-opts-value]: DELETE_COMPLETE  state changed
2025-10-11 12:16:20.021988 | hotstack | 2025-10-11 16:16:10Z [hs-918b704357334bf18129d9b4a9d53906.master2-cinder-vol1]: DELETE_COMPLETE  state changed
2025-10-11 12:16:20.022001 | hotstack | 2025-10-11 16:16:10Z [hs-918b704357334bf18129d9b4a9d53906.controller-machine-port]: DELETE_IN_PROGRESS  state changed
2025-10-11 12:16:20.022014 | hotstack | 2025-10-11 16:16:11Z [hs-918b704357334bf18129d9b4a9d53906.master2-trunk0]: DELETE_COMPLETE  state changed
2025-10-11 12:16:20.022027 | hotstack | 2025-10-11 16:16:11Z [hs-918b704357334bf18129d9b4a9d53906.master2-storage-port]: DELETE_IN_PROGRESS  state changed
2025-10-11 12:16:20.022041 | hotstack | 2025-10-11 16:16:11Z [hs-918b704357334bf18129d9b4a9d53906.master2-ctlplane-trunk-parent-port]: DELETE_IN_PROGRESS  state changed
2025-10-11 12:16:20.022054 | hotstack | 2025-10-11 16:16:11Z [hs-918b704357334bf18129d9b4a9d53906.ironic-subnet]: DELETE_COMPLETE  state changed
2025-10-11 12:16:20.022073 | hotstack | 2025-10-11 16:16:11Z [hs-918b704357334bf18129d9b4a9d53906.master2-tenant-port]: DELETE_IN_PROGRESS  state changed
2025-10-11 12:16:20.022086 | hotstack | 2025-10-11 16:16:11Z [hs-918b704357334bf18129d9b4a9d53906.master2-internal-api-port]: DELETE_IN_PROGRESS  state changed
2025-10-11 12:16:20.022099 | hotstack | 2025-10-11 16:16:11Z [hs-918b704357334bf18129d9b4a9d53906.master2-octavia-port]: DELETE_IN_PROGRESS  state changed
2025-10-11 12:16:20.022113 | hotstack | 2025-10-11 16:16:11Z [hs-918b704357334bf18129d9b4a9d53906.ironic-net]: DELETE_IN_PROGRESS  state changed
2025-10-11 12:16:20.022126 | hotstack | 2025-10-11 16:16:11Z [hs-918b704357334bf18129d9b4a9d53906.controller-machine-port]: DELETE_COMPLETE  state changed
2025-10-11 12:16:20.022139 | hotstack | 2025-10-11 16:16:11Z [hs-918b704357334bf18129d9b4a9d53906.machine-subnet]: DELETE_IN_PROGRESS  state changed
2025-10-11 12:16:20.022152 | hotstack | 2025-10-11 16:16:11Z [hs-918b704357334bf18129d9b4a9d53906.master2-cinder-vol0]: DELETE_COMPLETE  state changed
2025-10-11 12:16:20.022166 | hotstack | 2025-10-11 16:16:11Z [hs-918b704357334bf18129d9b4a9d53906.master2-storage-port]: DELETE_COMPLETE  state changed
2025-10-11 12:16:20.022203 | hotstack | 2025-10-11 16:16:11Z [hs-918b704357334bf18129d9b4a9d53906.master2-cinder-vol2]: DELETE_COMPLETE  state changed
2025-10-11 12:16:20.022222 | hotstack | 2025-10-11 16:16:12Z [hs-918b704357334bf18129d9b4a9d53906.master2-ctlplane-trunk-parent-port]: DELETE_COMPLETE  state changed
2025-10-11 12:16:20.022235 | hotstack | 2025-10-11 16:16:12Z [hs-918b704357334bf18129d9b4a9d53906.ctlplane-subnet]: DELETE_IN_PROGRESS  state changed
2025-10-11 12:16:20.022249 | hotstack | 2025-10-11 16:16:12Z [hs-918b704357334bf18129d9b4a9d53906.storage-subnet]: DELETE_IN_PROGRESS  state changed
2025-10-11 12:16:20.022262 | hotstack | 2025-10-11 16:16:12Z [hs-918b704357334bf18129d9b4a9d53906.master2-internal-api-port]: DELETE_COMPLETE  state changed
2025-10-11 12:16:20.022275 | hotstack | 2025-10-11 16:16:12Z [hs-918b704357334bf18129d9b4a9d53906.internal-api-subnet]: DELETE_IN_PROGRESS  state changed
2025-10-11 12:16:20.022288 | hotstack | 2025-10-11 16:16:12Z [hs-918b704357334bf18129d9b4a9d53906.machine-subnet]: DELETE_COMPLETE  state changed
2025-10-11 12:16:20.022301 | hotstack | 2025-10-11 16:16:12Z [hs-918b704357334bf18129d9b4a9d53906.machine-net]: DELETE_IN_PROGRESS  state changed
2025-10-11 12:16:20.022315 | hotstack | 2025-10-11 16:16:12Z [hs-918b704357334bf18129d9b4a9d53906.ctlplane-subnet]: DELETE_COMPLETE  state changed
2025-10-11 12:16:20.022337 | hotstack | 2025-10-11 16:16:12Z [hs-918b704357334bf18129d9b4a9d53906.master2-tenant-port]: DELETE_COMPLETE  state changed
2025-10-11 12:16:20.022350 | hotstack | 2025-10-11 16:16:12Z [hs-918b704357334bf18129d9b4a9d53906.ironic-net]: DELETE_COMPLETE  state changed
2025-10-11 12:16:20.022363 | hotstack | 2025-10-11 16:16:12Z [hs-918b704357334bf18129d9b4a9d53906.master2-octavia-port]: DELETE_COMPLETE  state changed
2025-10-11 12:16:20.022376 | hotstack | 2025-10-11 16:16:12Z [hs-918b704357334bf18129d9b4a9d53906.tenant-subnet]: DELETE_IN_PROGRESS  state changed
2025-10-11 12:16:20.022389 | hotstack | 2025-10-11 16:16:12Z [hs-918b704357334bf18129d9b4a9d53906.internal-api-subnet]: DELETE_COMPLETE  state changed
2025-10-11 12:16:20.022402 | hotstack | 2025-10-11 16:16:12Z [hs-918b704357334bf18129d9b4a9d53906.ctlplane-net]: DELETE_IN_PROGRESS  state changed
2025-10-11 12:16:20.022428 | hotstack | 2025-10-11 16:16:12Z [hs-918b704357334bf18129d9b4a9d53906.storage-subnet]: DELETE_COMPLETE  state changed
2025-10-11 12:16:20.112216 | hotstack | 2025-10-11 16:16:12Z [hs-918b704357334bf18129d9b4a9d53906.octavia-subnet]: DELETE_IN_PROGRESS  state changed
2025-10-11 12:16:20.112308 | hotstack | 2025-10-11 16:16:13Z [hs-918b704357334bf18129d9b4a9d53906.storage-net]: DELETE_IN_PROGRESS  state changed
2025-10-11 12:16:20.112457 | hotstack | 2025-10-11 16:16:13Z [hs-918b704357334bf18129d9b4a9d53906.internal-api-net]: DELETE_IN_PROGRESS  state changed
2025-10-11 12:16:20.112474 | hotstack | 2025-10-11 16:16:13Z [hs-918b704357334bf18129d9b4a9d53906.tenant-subnet]: DELETE_COMPLETE  state changed
2025-10-11 12:16:20.112488 | hotstack | 2025-10-11 16:16:13Z [hs-918b704357334bf18129d9b4a9d53906.tenant-net]: DELETE_IN_PROGRESS  state changed
2025-10-11 12:16:20.112501 | hotstack | 2025-10-11 16:16:13Z [hs-918b704357334bf18129d9b4a9d53906.octavia-subnet]: DELETE_COMPLETE  state changed
2025-10-11 12:16:20.112540 | hotstack | 2025-10-11 16:16:13Z [hs-918b704357334bf18129d9b4a9d53906.octavia-net]: DELETE_IN_PROGRESS  state changed
2025-10-11 12:16:20.112553 | hotstack | 2025-10-11 16:16:13Z [hs-918b704357334bf18129d9b4a9d53906.machine-net]: DELETE_COMPLETE  state changed
2025-10-11 12:16:20.112567 | hotstack | 2025-10-11 16:16:13Z [hs-918b704357334bf18129d9b4a9d53906.ctlplane-net]: DELETE_COMPLETE  state changed
2025-10-11 12:16:20.112580 | hotstack | 2025-10-11 16:16:14Z [hs-918b704357334bf18129d9b4a9d53906.internal-api-net]: DELETE_COMPLETE  state changed
2025-10-11 12:16:20.112593 | hotstack | 2025-10-11 16:16:14Z [hs-918b704357334bf18129d9b4a9d53906.storage-net]: DELETE_COMPLETE  state changed
2025-10-11 12:16:20.112606 | hotstack | 2025-10-11 16:16:14Z [hs-918b704357334bf18129d9b4a9d53906.tenant-net]: DELETE_COMPLETE  state changed
2025-10-11 12:16:20.112619 | hotstack | 2025-10-11 16:16:15Z [hs-918b704357334bf18129d9b4a9d53906.octavia-net]: DELETE_COMPLETE  state changed
2025-10-11 12:16:20.112632 | hotstack | 2025-10-11 16:16:15Z [hs-918b704357334bf18129d9b4a9d53906.master1-cinder-vol1]: DELETE_COMPLETE  state changed
2025-10-11 12:16:20.112645 | hotstack | 2025-10-11 16:16:15Z [hs-918b704357334bf18129d9b4a9d53906.master1-cinder-vol0]: DELETE_COMPLETE  state changed
2025-10-11 12:16:20.112658 | hotstack | 2025-10-11 16:16:15Z [hs-918b704357334bf18129d9b4a9d53906.master1-lvms-vol0]: DELETE_COMPLETE  state changed
2025-10-11 12:16:20.112671 | hotstack | 2025-10-11 16:16:15Z [hs-918b704357334bf18129d9b4a9d53906.master1-cinder-vol2]: DELETE_COMPLETE  state changed
2025-10-11 12:16:20.112685 | hotstack | 2025-10-11 16:16:15Z [hs-918b704357334bf18129d9b4a9d53906]: DELETE_COMPLETE  Stack DELETE completed successfully
2025-10-11 16:16:20.160717 | hotstack | changed
2025-10-11 16:16:20.174557 | 
2025-10-11 16:16:20.174735 | TASK [Print message that stack failed to delete]
2025-10-11 16:16:20.200707 | hotstack | skipping: Conditional result was False
2025-10-11 16:16:20.212160 | 
2025-10-11 16:16:20.212286 | TASK [Print message that stack deleted successfully]
2025-10-11 16:16:20.259719 | hotstack | ok: Stack deleted successfully.
2025-10-11 16:16:20.270115 | 
2025-10-11 16:16:20.270197 | TASK [Delete clouds.yaml]
2025-10-11 16:16:20.581513 | hotstack | Output suppressed because no_log was given
2025-10-11 16:16:20.619549 | 
2025-10-11 16:16:20.619645 | PLAY RECAP
2025-10-11 16:16:20.619732 | hotstack | ok: 18 changed: 8 unreachable: 0 failed: 0 skipped: 3 rescued: 2 ignored: 1
2025-10-11 16:16:20.619766 | 
2025-10-11 16:16:20.731576 | POST-RUN END RESULT_NORMAL: [trusted : review.rdoproject.org/config/playbooks/hotstack/post.yaml@master]
2025-10-11 16:16:20.734316 | POST-RUN START: [trusted : review.rdoproject.org/config/playbooks/base-minimal/post-ssh.yaml@master]
2025-10-11 16:16:21.316175 | 
2025-10-11 16:16:21.316284 | PLAY [all]
2025-10-11 16:16:21.335045 | 
2025-10-11 16:16:21.335118 | TASK [include_role : fetch-output]
2025-10-11 16:16:21.372992 | hotstack | ok
2025-10-11 16:16:21.390745 | 
2025-10-11 16:16:21.390820 | TASK [fetch-output : Set log path for multiple nodes]
2025-10-11 16:16:21.434652 | hotstack | skipping: Conditional result was False
2025-10-11 16:16:21.440715 | 
2025-10-11 16:16:21.440786 | TASK [fetch-output : Set log path for single node]
2025-10-11 16:16:21.480062 | hotstack | ok
2025-10-11 16:16:21.484974 | 
2025-10-11 16:16:21.485039 | LOOP [fetch-output : Ensure local output dirs]
2025-10-11 16:16:21.874906 | hotstack -> localhost | ok: "/var/lib/zuul/builds/918b704357334bf18129d9b4a9d53906/work/logs"
2025-10-11 16:16:22.070625 | hotstack -> localhost | changed: "/var/lib/zuul/builds/918b704357334bf18129d9b4a9d53906/work/artifacts"
2025-10-11 16:16:22.273994 | hotstack -> localhost | changed: "/var/lib/zuul/builds/918b704357334bf18129d9b4a9d53906/work/docs"
2025-10-11 16:16:22.301123 | 
2025-10-11 16:16:22.301263 | LOOP [fetch-output : Collect logs, artifacts and docs]
2025-10-11 16:16:23.073219 | hotstack | changed:
2025-10-11 16:16:23.077948 | hotstack | .d..t...... ./
2025-10-11 16:16:23.078106 | hotstack | cd+++++++++ hotlogs/
2025-10-11 16:16:23.078158 | hotstack | cd+++++++++ hotlogs/data/
2025-10-11 16:16:23.078201 | hotstack | cd+++++++++ hotlogs/manifests/
2025-10-11 16:16:23.078250 | hotstack | cd+++++++++ hotlogs/ocp_cluster/
2025-10-11 16:16:23.078290 | hotstack | >f+++++++++ hotlogs/ocp_cluster/openshift_install.log
2025-10-11 16:16:23.078329 | hotstack | cd+++++++++ hotlogs/ocp_cluster/butane/
2025-10-11 16:16:23.078367 | hotstack | >f+++++++++ hotlogs/ocp_cluster/butane/90-master-disable-netifnames.bu
2025-10-11 16:16:23.078407 | hotstack | >f+++++++++ hotlogs/ocp_cluster/butane/90-master-enable-iscsi.bu
2025-10-11 16:16:23.078444 | hotstack | >f+++++++++ hotlogs/ocp_cluster/butane/91-master-enable-multipath.bu
2025-10-11 16:16:23.078480 | hotstack | >f+++++++++ hotlogs/ocp_cluster/butane/92-master-lv-cinder-volumes.bu
2025-10-11 16:16:23.078517 | hotstack | cd+++++++++ hotlogs/ocp_cluster/config-assets/
2025-10-11 16:16:23.078554 | hotstack | >f+++++++++ hotlogs/ocp_cluster/config-assets/95-etcd_config.yaml
2025-10-11 16:16:23.078589 | hotstack | >f+++++++++ hotlogs/ocp_cluster/config-assets/ovn_k8s_config.yaml
2025-10-11 16:16:23.078625 | hotstack | cd+++++++++ hotlogs/ocp_cluster/machine-configs/
2025-10-11 16:16:23.078667 | hotstack | >f+++++++++ hotlogs/ocp_cluster/machine-configs/90-master-disable-netifnames.yaml
2025-10-11 16:16:23.078770 | hotstack | >f+++++++++ hotlogs/ocp_cluster/machine-configs/90-master-enable-iscsi.yaml
2025-10-11 16:16:23.078811 | hotstack | >f+++++++++ hotlogs/ocp_cluster/machine-configs/91-master-enable-multipath.yaml
2025-10-11 16:16:23.078847 | hotstack | >f+++++++++ hotlogs/ocp_cluster/machine-configs/92-master-lv-cinder-volumes.yaml
2025-10-11 16:16:23.078939 | hotstack | changed: All items complete
2025-10-11 16:16:23.078983 | 
2025-10-11 16:16:23.642896 | hotstack | changed: .d..t...... ./
2025-10-11 16:16:24.175249 | hotstack | changed: .d..t...... ./
2025-10-11 16:16:24.205958 | 
2025-10-11 16:16:24.206113 | TASK [include_role : fetch-output-openshift]
2025-10-11 16:16:24.233227 | hotstack | skipping: Conditional result was False
2025-10-11 16:16:24.245822 | 
2025-10-11 16:16:24.245957 | LOOP [merge-output-to-logs : Move artifacts and docs to logs dir]
2025-10-11 16:16:24.683858 | hotstack -> localhost | ok: Item: artifacts Runtime: 0:00:00.008915
2025-10-11 16:16:24.903280 | hotstack -> localhost | ok: Item: docs Runtime: 0:00:00.012257
2025-10-11 16:16:24.959141 | 
2025-10-11 16:16:24.959287 | PLAY [all]
2025-10-11 16:16:24.980359 | 
2025-10-11 16:16:24.980449 | TASK [remove-build-sshkey : Remove the build SSH key from all nodes]
2025-10-11 16:16:25.681354 | hotstack | changed
2025-10-11 16:16:25.726332 | 
2025-10-11 16:16:25.726427 | PLAY RECAP
2025-10-11 16:16:25.726483 | hotstack | ok: 5 changed: 4 unreachable: 0 failed: 0 skipped: 2 rescued: 0 ignored: 0
2025-10-11 16:16:25.726516 | 
2025-10-11 16:16:25.831956 | POST-RUN END RESULT_NORMAL: [trusted : review.rdoproject.org/config/playbooks/base-minimal/post-ssh.yaml@master]
2025-10-11 16:16:25.834611 | POST-RUN START: [trusted : review.rdoproject.org/config/playbooks/base-minimal/post-logs.yaml@master]
2025-10-11 16:16:26.500909 | 
2025-10-11 16:16:26.501028 | PLAY [localhost]
2025-10-11 16:16:26.518520 | 
2025-10-11 16:16:26.518594 | TASK [generate-zuul-manifest : Generate Zuul manifest]
2025-10-11 16:16:26.910885 | localhost | changed
2025-10-11 16:16:26.918435 | 
2025-10-11 16:16:26.918591 | TASK [generate-zuul-manifest : Return Zuul manifest URL to Zuul]
2025-10-11 16:16:26.949521 | localhost | ok
2025-10-11 16:16:26.960205 | 
2025-10-11 16:16:26.960307 | TASK [add-fileserver : Create SSH private key tempfile]
2025-10-11 16:16:27.370273 | localhost | changed
2025-10-11 16:16:27.381899 | 
2025-10-11 16:16:27.382036 | TASK [add-fileserver : Create SSH private key from secret]
2025-10-11 16:16:28.143738 | localhost | changed
2025-10-11 16:16:28.148754 | 
2025-10-11 16:16:28.148817 | TASK [add-fileserver : Add fileserver ssh key]
2025-10-11 16:16:28.619268 | localhost | Identity added: /var/lib/zuul/builds/918b704357334bf18129d9b4a9d53906/work/tmp/ansible.t1p9c7uh (/var/lib/zuul/builds/918b704357334bf18129d9b4a9d53906/work/tmp/ansible.t1p9c7uh)
2025-10-11 16:16:28.619558 | localhost | ok: Runtime: 0:00:00.012243
2025-10-11 16:16:28.628839 | 
2025-10-11 16:16:28.628976 | TASK [add-fileserver : Remove SSH private key from disk]
2025-10-11 16:16:28.939556 | localhost | ok: Runtime: 0:00:00.005796
2025-10-11 16:16:28.949584 | 
2025-10-11 16:16:28.949746 | TASK [add-fileserver : Add fileserver to inventory]
2025-10-11 16:16:29.006786 | localhost | changed
2025-10-11 16:16:29.015642 | 
2025-10-11 16:16:29.015804 | TASK [add-fileserver : Add fileserver server to known hosts]
2025-10-11 16:16:29.414119 | localhost | changed
2025-10-11 16:16:29.440842 | 
2025-10-11 16:16:29.440916 | PLAY [localhost]
2025-10-11 16:16:29.457551 | 
2025-10-11 16:16:29.457612 | TASK [Generate bulk log download script]
2025-10-11 16:16:29.503342 | localhost | ok
2025-10-11 16:16:29.515271 | 
2025-10-11 16:16:29.515344 | TASK [local-log-download : Check API endpoint is defined]
2025-10-11 16:16:29.543371 | localhost | ok: All assertions passed
2025-10-11 16:16:29.547781 | 
2025-10-11 16:16:29.547854 | TASK [local-log-download : Create download script]
2025-10-11 16:16:29.998369 | localhost -> localhost | changed
2025-10-11 16:16:30.016585 | 
2025-10-11 16:16:30.016744 | TASK [Register quick-download link]
2025-10-11 16:16:30.042638 | localhost | ok
2025-10-11 16:16:30.073664 | 
2025-10-11 16:16:30.073762 | PLAY [logserver.rdoproject.org]
2025-10-11 16:16:30.087210 | 
2025-10-11 16:16:30.087284 | TASK [Set zuul-log-path fact]
2025-10-11 16:16:30.105380 | logserver.rdoproject.org | ok
2025-10-11 16:16:30.117140 | 
2025-10-11 16:16:30.117242 | TASK [set-zuul-log-path-fact : Set log path for a build]
2025-10-11 16:16:30.145190 | logserver.rdoproject.org | ok
2025-10-11 16:16:30.152587 | 
2025-10-11 16:16:30.152670 | TASK [upload-logs : Create log directories]
2025-10-11 16:16:30.804177 | logserver.rdoproject.org | changed
2025-10-11 16:16:30.810769 | 
2025-10-11 16:16:30.810899 | TASK [upload-logs : Ensure logs are readable before uploading]
2025-10-11 16:16:31.102671 | logserver.rdoproject.org -> localhost | ok: Runtime: 0:00:00.008914
2025-10-11 16:16:31.112105 | 
2025-10-11 16:16:31.112237 | TASK [upload-logs : Upload logs to log server]
2025-10-11 16:16:31.827393 | logserver.rdoproject.org | Output suppressed because no_log was given
2025-10-11 16:16:31.833952 | 
2025-10-11 16:16:31.834089 | LOOP [upload-logs : Compress console log and json output]
2025-10-11 16:16:31.904704 | logserver.rdoproject.org | skipping: Conditional result was False
2025-10-11 16:16:31.915814 | logserver.rdoproject.org | skipping: Conditional result was False
2025-10-11 16:16:31.929602 | 
2025-10-11 16:16:31.929815 | LOOP [upload-logs : Upload compressed console log and json output]
2025-10-11 16:16:31.980779 | logserver.rdoproject.org | skipping: Conditional result was False
2025-10-11 16:16:31.981176 | 
2025-10-11 16:16:31.983882 | logserver.rdoproject.org | skipping: Conditional result was False
2025-10-11 16:16:31.995640 | 
2025-10-11 16:16:31.995858 | LOOP [upload-logs : Upload console log and json output]