2026-03-08 22:23:53.372404 | Job console starting...
2026-03-08 22:23:53.387412 | Updating repositories
2026-03-08 22:23:53.414556 | Preparing job workspace
2026-03-08 22:23:59.708995 | Running Ansible setup...
2026-03-08 22:24:07.242157 | PRE-RUN START: [trusted : review.rdoproject.org/config/playbooks/base-minimal/pre.yaml@master]
2026-03-08 22:24:07.801720 | 
2026-03-08 22:24:07.801838 | PLAY [localhost]
2026-03-08 22:24:07.810344 | 
2026-03-08 22:24:07.810413 | TASK [Gathering Facts]
2026-03-08 22:24:08.804485 | localhost | ok
2026-03-08 22:24:08.831971 | 
2026-03-08 22:24:08.832137 | TASK [log-inventory : Ensure Zuul Ansible directory exists]
2026-03-08 22:24:09.255754 | localhost -> localhost | changed
2026-03-08 22:24:09.264252 | 
2026-03-08 22:24:09.264355 | TASK [log-inventory : Copy ansible inventory to logs dir]
2026-03-08 22:24:10.166443 | localhost -> localhost | changed
2026-03-08 22:24:10.185264 | 
2026-03-08 22:24:10.185397 | TASK [Setup log path fact]
2026-03-08 22:24:10.210134 | localhost | ok
2026-03-08 22:24:10.235393 | 
2026-03-08 22:24:10.235538 | TASK [set-zuul-log-path-fact : Set log path for a build]
2026-03-08 22:24:10.275444 | localhost | ok
2026-03-08 22:24:10.286190 | 
2026-03-08 22:24:10.286278 | TASK [emit-job-header : Print job information]
2026-03-08 22:24:10.339668 | # Job Information
2026-03-08 22:24:10.339989 | Ansible Version: 2.15.12
2026-03-08 22:24:10.340077 | Job: vexxhost-hotstack-3-nodes-pod-test
2026-03-08 22:24:10.340124 | Pipeline: github-check
2026-03-08 22:24:10.340167 | Executor: ze03.softwarefactory-project.io
2026-03-08 22:24:10.340206 | Triggered by: https://github.com/openstack-k8s-operators/hotstack/pull/404
2026-03-08 22:24:10.340246 | Log URL (when completed): https://logserver.rdoproject.org/661/rdoproject.org/66191bf4813c41419926694e7f63197e/
2026-03-08 22:24:10.340284 | Event ID: 76dbd730-1b3d-11f1-8dd3-c916d2ba081a
2026-03-08 22:24:10.348359 | 
2026-03-08 22:24:10.348494 | LOOP [emit-job-header : Print node information]
2026-03-08 22:24:10.465207 | localhost | ok:
2026-03-08 22:24:10.465576 | localhost | # Node Information
2026-03-08 22:24:10.465637 | localhost | Inventory Hostname: hotstack
2026-03-08 22:24:10.465689 | localhost | Hostname: 3bad4a19298b
2026-03-08 22:24:10.465769 | localhost | Username: zuul
2026-03-08 22:24:10.465826 | localhost | Distro: CentOS 9
2026-03-08 22:24:10.465868 | localhost | Provider: k1s04
2026-03-08 22:24:10.465907 | localhost | Region:
2026-03-08 22:24:10.465944 | localhost | Label: pod-centos-9-stream
2026-03-08 22:24:10.465981 | localhost | Product Name: OpenStack Nova
2026-03-08 22:24:10.466019 | localhost | Interface IP: pod-centos-9-stream-0005641844
2026-03-08 22:24:10.508118 | 
2026-03-08 22:24:10.508267 | PLAY [all]
2026-03-08 22:24:10.521980 | 
2026-03-08 22:24:10.522112 | TASK [Gather network facts]
2026-03-08 22:24:16.467010 | hotstack | ok
2026-03-08 22:24:16.493377 | 
2026-03-08 22:24:16.493493 | TASK [include_role : start-zuul-console]
2026-03-08 22:24:16.514458 | hotstack | ok
2026-03-08 22:24:16.530031 | 
2026-03-08 22:24:16.530148 | TASK [start-zuul-console : Start zuul_console daemon.]
2026-03-08 22:24:22.174218 | hotstack | ok
2026-03-08 22:24:22.193898 | 
2026-03-08 22:24:22.194088 | TASK [include_role : add-build-sshkey]
2026-03-08 22:24:22.221123 | hotstack | skipping: Conditional result was False
2026-03-08 22:24:22.231434 | 
2026-03-08 22:24:22.231563 | TASK [include_role : validate-host]
2026-03-08 22:24:22.257980 | hotstack | skipping: Conditional result was False
2026-03-08 22:24:22.268060 | 
2026-03-08 22:24:22.268195 | TASK [include_role : prepare-workspace-openshift]
2026-03-08 22:24:22.303050 | hotstack | ok
2026-03-08 22:24:22.327153 | 
2026-03-08 22:24:22.327249 | LOOP [prepare-workspace-openshift : include_tasks]
2026-03-08 22:24:22.400797 | hotstack | ok:
2026-03-08 22:24:22.401162 | hotstack | [
2026-03-08 22:24:22.401217 | hotstack |   "hotstack",
2026-03-08 22:24:22.401262 | hotstack |   {
2026-03-08 22:24:22.401299 | hotstack |     "context": "main:zuul-worker/38-102-83-54:9023",
2026-03-08 22:24:22.401335 | hotstack |     "namespace": "main",
2026-03-08 22:24:22.401390 | hotstack |     "pod": "pod-centos-9-stream-0005641844",
2026-03-08 22:24:22.401425 | hotstack |     "stream_port": "44583"
2026-03-08 22:24:22.401459 | hotstack |   }
2026-03-08 22:24:22.401492 | hotstack | ]
2026-03-08 22:24:22.401554 | hotstack | ok: All items complete
2026-03-08 22:24:22.401596 | 
2026-03-08 22:24:22.410986 | hotstack | included: /var/lib/zuul/builds/66191bf4813c41419926694e7f63197e/trusted/project_0/review.rdoproject.org/config/roles/prepare-workspace-openshift/tasks/rsync.yaml
2026-03-08 22:24:22.418021 | 
2026-03-08 22:24:22.418126 | TASK [prepare-workspace-openshift : Create src directory]
2026-03-08 22:24:23.758647 | hotstack -> localhost | ok: Runtime: 0:00:00.900624
2026-03-08 22:24:23.803558 | 
2026-03-08 22:24:23.803690 | TASK [prepare-workspace-openshift : Copy src repos to the pod]
2026-03-08 22:24:26.332169 | hotstack -> localhost | Output suppressed because no_log was given
2026-03-08 22:24:26.353959 | 
2026-03-08 22:24:26.354103 | TASK [include_role : remove-zuul-sshkey]
2026-03-08 22:24:26.383946 | hotstack | ok
2026-03-08 22:24:26.411496 | 
2026-03-08 22:24:26.411584 | TASK [remove-zuul-sshkey : Remove master key from local agent]
2026-03-08 22:24:26.834247 | hotstack -> localhost | changed
2026-03-08 22:24:26.854763 | 
2026-03-08 22:24:26.854906 | LOOP [ensure-output-dirs : Empty Zuul Output directories by removing them]
2026-03-08 22:24:32.624517 | hotstack | ok: "logs"
2026-03-08 22:24:38.406996 | hotstack | ok: "artifacts"
2026-03-08 22:24:44.200090 | hotstack | ok: "docs"
2026-03-08 22:24:44.213666 | 
2026-03-08 22:24:44.213819 | LOOP [ensure-output-dirs : Ensure Zuul Output directories exist]
2026-03-08 22:24:49.834069 | hotstack | changed: "logs"
2026-03-08 22:24:49.834472 | hotstack | ok: All items complete
2026-03-08 22:24:49.834529 | 
2026-03-08 22:24:55.924220 | hotstack | changed: "artifacts"
2026-03-08 22:25:01.722455 | hotstack | changed: "docs"
2026-03-08 22:25:01.777391 | 
2026-03-08 22:25:01.777519 | PLAY RECAP
2026-03-08 22:25:01.777578 | hotstack | ok: 8 changed: 4 unreachable: 0 failed: 0 skipped: 2 rescued: 0 ignored: 0
2026-03-08 22:25:01.777616 | localhost | ok: 6 changed: 2 unreachable: 0 failed: 0 skipped: 0 rescued: 0 ignored: 0
2026-03-08 22:25:01.777642 | 
2026-03-08 22:25:01.902473 | PRE-RUN END RESULT_NORMAL: [trusted : review.rdoproject.org/config/playbooks/base-minimal/pre.yaml@master]
2026-03-08 22:25:01.907336 | PRE-RUN START: [trusted : review.rdoproject.org/config/playbooks/hotstack/pre_stage1.yaml@master]
2026-03-08 22:25:02.523268 | 
2026-03-08 22:25:02.523384 | PLAY [Create necessary files to deploy hotstack scenario]
2026-03-08 22:25:02.542123 | 
2026-03-08 22:25:02.542205 | TASK [prepare-workspace : Start zuul_console daemon.]
2026-03-08 22:25:08.378958 | hotstack | ok
2026-03-08 22:25:08.385145 | 
2026-03-08 22:25:08.385229 | TASK [prepare-workspace : Synchronize src repos to workspace directory.]
2026-03-08 22:25:08.424425 | hotstack | Output suppressed because no_log was given
failure
2026-03-08 22:25:08.425777 | 
2026-03-08 22:25:08.425832 | PLAY RECAP
2026-03-08 22:25:08.425881 | hotstack | ok: 1 changed: 0 unreachable: 0 failed: 1 skipped: 0 rescued: 0 ignored: 0
2026-03-08 22:25:08.425908 | 
2026-03-08 22:25:08.512088 | PRE-RUN END RESULT_NORMAL: [trusted : review.rdoproject.org/config/playbooks/hotstack/pre_stage1.yaml@master]
2026-03-08 22:25:08.516821 | POST-RUN START: [trusted : review.rdoproject.org/config/playbooks/hotstack/post.yaml@master]
2026-03-08 22:25:09.097767 | 
2026-03-08 22:25:09.097877 | PLAY [Collect hotstack logs and clean up environment]
2026-03-08 22:25:09.118080 | 
2026-03-08 22:25:09.118160 | TASK [Fetch stack outputs from remote to zuul executor]
2026-03-08 22:25:19.175947 | hotstack | Output suppressed because no_log was given
2026-03-08 22:25:19.180425 | 
2026-03-08 22:25:19.180499 | TASK [Print message that log collection failed]
2026-03-08 22:25:19.208377 | hotstack | ok: WARNING: Log collection failed and may be incomplete. Continue with cleanup ...
2026-03-08 22:25:19.212756 | 
2026-03-08 22:25:19.212814 | TASK [Set _auth_type fact]
2026-03-08 22:25:19.250445 | hotstack | Output suppressed because no_log was given
2026-03-08 22:25:19.254895 | 
2026-03-08 22:25:19.254954 | TASK [Write clouds.yaml]
2026-03-08 22:25:29.984883 | hotstack | Output suppressed because no_log was given
failure
2026-03-08 22:25:29.986233 | 
2026-03-08 22:25:29.986408 | PLAY RECAP
2026-03-08 22:25:29.986520 | hotstack | ok: 2 changed: 0 unreachable: 0 failed: 1 skipped: 0 rescued: 1 ignored: 0
2026-03-08 22:25:29.986581 | 
2026-03-08 22:25:30.098213 | POST-RUN END RESULT_NORMAL: [trusted : review.rdoproject.org/config/playbooks/hotstack/post.yaml@master]
2026-03-08 22:25:30.101063 | POST-RUN START: [trusted : review.rdoproject.org/config/playbooks/base-minimal/post-ssh.yaml@master]
2026-03-08 22:25:30.649656 | 
2026-03-08 22:25:30.649796 | PLAY [all]
2026-03-08 22:25:30.667199 | 
2026-03-08 22:25:30.667277 | TASK [include_role : fetch-output]
2026-03-08 22:25:30.691144 | hotstack | skipping: Conditional result was False
2026-03-08 22:25:30.696222 | 
2026-03-08 22:25:30.696297 | TASK [include_role : fetch-output-openshift]
2026-03-08 22:25:30.734542 | hotstack | ok
2026-03-08 22:25:30.749215 | 
2026-03-08 22:25:30.749278 | TASK [fetch-output-openshift : Set log path for multiple nodes]
2026-03-08 22:25:30.803943 | hotstack | skipping: Conditional result was False
2026-03-08 22:25:30.809944 | 
2026-03-08 22:25:30.810021 | TASK [fetch-output-openshift : Set log path for single node]
2026-03-08 22:25:30.853548 | hotstack | ok
2026-03-08 22:25:30.867300 | 
2026-03-08 22:25:30.867466 | LOOP [fetch-output-openshift : Ensure local output dirs]
2026-03-08 22:25:31.277268 | hotstack -> localhost | ok: "/var/lib/zuul/builds/66191bf4813c41419926694e7f63197e/work/logs"
2026-03-08 22:25:31.478561 | hotstack -> localhost | changed: "/var/lib/zuul/builds/66191bf4813c41419926694e7f63197e/work/logs/npm"
2026-03-08 22:25:31.730285 | hotstack -> localhost | changed: "/var/lib/zuul/builds/66191bf4813c41419926694e7f63197e/work/artifacts"
2026-03-08 22:25:31.993528 | hotstack -> localhost | changed: "/var/lib/zuul/builds/66191bf4813c41419926694e7f63197e/work/docs"
2026-03-08 22:25:32.012892 | 
2026-03-08 22:25:32.013081 | LOOP [fetch-output-openshift : include_tasks]
2026-03-08 22:25:32.096607 | hotstack | ok:
2026-03-08 22:25:32.097018 | hotstack | [
2026-03-08 22:25:32.097077 | hotstack |   "hotstack",
2026-03-08 22:25:32.097119 | hotstack |   {
2026-03-08 22:25:32.097157 | hotstack |     "context": "main:zuul-worker/38-102-83-54:9023",
2026-03-08 22:25:32.097197 | hotstack |     "namespace": "main",
2026-03-08 22:25:32.097235 | hotstack |     "pod": "pod-centos-9-stream-0005641844",
2026-03-08 22:25:32.097272 | hotstack |     "stream_port": "44583"
2026-03-08 22:25:32.097307 | hotstack |   }
2026-03-08 22:25:32.097343 | hotstack | ]
2026-03-08 22:25:32.097411 | hotstack | changed: All items complete
2026-03-08 22:25:32.097455 | 
2026-03-08 22:25:32.118354 | hotstack | included: /var/lib/zuul/builds/66191bf4813c41419926694e7f63197e/trusted/project_0/review.rdoproject.org/config/roles/fetch-output-openshift/tasks/rsync.yaml
2026-03-08 22:25:32.131434 | 
2026-03-08 22:25:32.131571 | LOOP [fetch-output-openshift : Ensure local directory exists]
2026-03-08 22:25:37.751804 | hotstack | changed: "/var/lib/zuul/builds/66191bf4813c41419926694e7f63197e/work/logs"
2026-03-08 22:25:43.481241 | hotstack | changed: "/var/lib/zuul/builds/66191bf4813c41419926694e7f63197e/work/logs/npm"
2026-03-08 22:25:49.029862 | hotstack | changed: "/var/lib/zuul/builds/66191bf4813c41419926694e7f63197e/work/artifacts"
2026-03-08 22:25:54.809429 | hotstack | changed: "/var/lib/zuul/builds/66191bf4813c41419926694e7f63197e/work/docs"
2026-03-08 22:25:54.833992 | 
2026-03-08 22:25:54.834183 | LOOP [fetch-output-openshift : Copy zuul-output from the pod to the executor]
2026-03-08 22:26:05.026509 | hotstack | Output suppressed because no_log was given
2026-03-08 22:26:05.032894 | 
2026-03-08 22:26:05.032965 | TASK [fetch-output-openshift : Remove empty directory]
2026-03-08 22:26:05.451637 | hotstack -> localhost | ok: Runtime: 0:00:00.066976
2026-03-08 22:26:05.484574 | 
2026-03-08 22:26:05.484880 | LOOP [merge-output-to-logs : Move artifacts and docs to logs dir]
2026-03-08 22:26:05.817471 | hotstack -> localhost | find: ‘/var/lib/zuul/builds/66191bf4813c41419926694e7f63197e/work/artifacts’: No such file or directory
2026-03-08 22:26:05.817819 | hotstack -> localhost | find: ‘/var/lib/zuul/builds/66191bf4813c41419926694e7f63197e/work/artifacts’: No such file or directory
2026-03-08 22:26:05.817879 | hotstack -> localhost | ok: Item: artifacts Runtime: 0:00:00.014523
2026-03-08 22:26:05.817918 | hotstack -> localhost | changed: All items complete
2026-03-08 22:26:05.817938 | 
2026-03-08 22:26:06.126250 | hotstack -> localhost | find: ‘/var/lib/zuul/builds/66191bf4813c41419926694e7f63197e/work/docs’: No such file or directory
2026-03-08 22:26:06.126444 | hotstack -> localhost | find: ‘/var/lib/zuul/builds/66191bf4813c41419926694e7f63197e/work/docs’: No such file or directory
2026-03-08 22:26:06.126478 | hotstack -> localhost | ok: Item: docs Runtime: 0:00:00.016480
2026-03-08 22:26:06.179859 | 
2026-03-08 22:26:06.180028 | PLAY [all]
2026-03-08 22:26:06.198823 | 
2026-03-08 22:26:06.198923 | TASK [remove-build-sshkey : Remove the build SSH key from all nodes]
2026-03-08 22:26:06.231453 | hotstack | ERROR
2026-03-08 22:26:06.231720 | hotstack | {
2026-03-08 22:26:06.231784 | hotstack |   "msg": "The 'file' lookup had an issue accessing the file '/var/lib/zuul/builds/66191bf4813c41419926694e7f63197e/work/src/../66191bf4813c41419926694e7f63197e_id_rsa.pub'. file not found, use -vvvvv to see paths searched"
2026-03-08 22:26:06.231816 | hotstack | }
2026-03-08 22:26:06.231853 | hotstack | ERROR: Ignoring Errors
2026-03-08 22:26:06.266928 | 
2026-03-08 22:26:06.267020 | PLAY RECAP
2026-03-08 22:26:06.267060 | hotstack | ok: 8 changed: 5 unreachable: 0 failed: 0 skipped: 2 rescued: 0 ignored: 1
2026-03-08 22:26:06.267080 | 
2026-03-08 22:26:06.370056 | POST-RUN END RESULT_NORMAL: [trusted : review.rdoproject.org/config/playbooks/base-minimal/post-ssh.yaml@master]
2026-03-08 22:26:06.374334 | POST-RUN START: [trusted : review.rdoproject.org/config/playbooks/base-minimal/post-logs.yaml@master]
2026-03-08 22:26:07.016255 | 
2026-03-08 22:26:07.016372 | PLAY [localhost]
2026-03-08 22:26:07.033897 | 
2026-03-08 22:26:07.033980 | TASK [generate-zuul-manifest : Generate Zuul manifest]
2026-03-08 22:26:07.415360 | localhost | changed
2026-03-08 22:26:07.425571 | 
2026-03-08 22:26:07.425765 | TASK [generate-zuul-manifest : Return Zuul manifest URL to Zuul]
2026-03-08 22:26:07.452292 | localhost | ok
2026-03-08 22:26:07.476161 | 
2026-03-08 22:26:07.476365 | TASK [add-fileserver : Create SSH private key tempfile]
2026-03-08 22:26:07.847453 | localhost | changed
2026-03-08 22:26:07.857689 | 
2026-03-08 22:26:07.857892 | TASK [add-fileserver : Create SSH private key from secret]
2026-03-08 22:26:08.561057 | localhost | changed
2026-03-08 22:26:08.565969 | 
2026-03-08 22:26:08.566030 | TASK [add-fileserver : Add fileserver ssh key]
2026-03-08 22:26:08.986249 | localhost | Identity added: /var/lib/zuul/builds/66191bf4813c41419926694e7f63197e/work/tmp/ansible.766f_rwx (/var/lib/zuul/builds/66191bf4813c41419926694e7f63197e/work/tmp/ansible.766f_rwx)
2026-03-08 22:26:08.986554 | localhost | ok: Runtime: 0:00:00.007854
2026-03-08 22:26:08.995948 | 
2026-03-08 22:26:08.996084 | TASK [add-fileserver : Remove SSH private key from disk]
2026-03-08 22:26:09.313754 | localhost | ok: Runtime: 0:00:00.006110
2026-03-08 22:26:09.320078 | 
2026-03-08 22:26:09.320163 | TASK [add-fileserver : Add fileserver to inventory]
2026-03-08 22:26:09.385347 | localhost | changed
2026-03-08 22:26:09.394553 | 
2026-03-08 22:26:09.394685 | TASK [add-fileserver : Add fileserver server to known hosts]
2026-03-08 22:26:09.806477 | localhost | changed
2026-03-08 22:26:09.831953 | 
2026-03-08 22:26:09.832041 | PLAY [localhost]
2026-03-08 22:26:09.847117 | 
2026-03-08 22:26:09.847223 | TASK [Generate bulk log download script]
2026-03-08 22:26:09.866889 | localhost | ok
2026-03-08 22:26:09.879438 | 
2026-03-08 22:26:09.879503 | TASK [local-log-download : Check API endpoint is defined]
2026-03-08 22:26:09.947610 | localhost | ok: All assertions passed
2026-03-08 22:26:09.954026 | 
2026-03-08 22:26:09.954109 | TASK [local-log-download : Create download script]
2026-03-08 22:26:10.385400 | localhost -> localhost | changed
2026-03-08 22:26:10.394062 | 
2026-03-08 22:26:10.394124 | TASK [Register quick-download link]
2026-03-08 22:26:10.411806 | localhost | ok
2026-03-08 22:26:10.427594 | 
2026-03-08 22:26:10.427649 | PLAY [logserver.rdoproject.org]
2026-03-08 22:26:10.437977 | 
2026-03-08 22:26:10.438033 | TASK [Set zuul-log-path fact]
2026-03-08 22:26:10.453865 | logserver.rdoproject.org | ok
2026-03-08 22:26:10.462503 | 
2026-03-08 22:26:10.462563 | TASK [set-zuul-log-path-fact : Set log path for a build]
2026-03-08 22:26:10.499688 | logserver.rdoproject.org | ok
2026-03-08 22:26:10.506916 | 
2026-03-08 22:26:10.506999 | TASK [upload-logs : Create log directories]
2026-03-08 22:26:11.326092 | logserver.rdoproject.org | changed
2026-03-08 22:26:11.344175 | 
2026-03-08 22:26:11.344343 | TASK [upload-logs : Ensure logs are readable before uploading]
2026-03-08 22:26:11.628059 | logserver.rdoproject.org -> localhost | ok: Runtime: 0:00:00.004782
2026-03-08 22:26:11.632814 | 
2026-03-08 22:26:11.632912 | TASK [upload-logs : Upload logs to log server]
2026-03-08 22:26:12.333656 | logserver.rdoproject.org | Output suppressed because no_log was given
2026-03-08 22:26:12.340213 | 
2026-03-08 22:26:12.340344 | LOOP [upload-logs : Compress console log and json output]
2026-03-08 22:26:12.411698 | logserver.rdoproject.org | skipping: Conditional result was False
2026-03-08 22:26:12.422841 | logserver.rdoproject.org | skipping: Conditional result was False
2026-03-08 22:26:12.435374 | 
2026-03-08 22:26:12.435567 | LOOP [upload-logs : Upload compressed console log and json output]
2026-03-08 22:26:12.484245 | logserver.rdoproject.org | skipping: Conditional result was False
2026-03-08 22:26:12.484643 | 
2026-03-08 22:26:12.487249 | logserver.rdoproject.org | skipping: Conditional result was False
2026-03-08 22:26:12.502381 | 
2026-03-08 22:26:12.502542 | LOOP [upload-logs : Upload console log and json output]