2026-02-02 11:23:55,175 p=36171 u=zuul n=ansible | [WARNING]: Found variable using reserved name: namespace 2026-02-02 11:23:55,175 p=36171 u=zuul n=ansible | PLAY [Execute the default-telemetry scenario] ********************************** 2026-02-02 11:23:55,209 p=36171 u=zuul n=ansible | TASK [default-telemetry : Check telemetry without COO and without Autoscaling and MetricStorage enabled] *** 2026-02-02 11:23:55,209 p=36171 u=zuul n=ansible | Monday 02 February 2026 11:23:55 +0000 (0:00:00.050) 0:00:00.050 ******* 2026-02-02 11:23:55,236 p=36171 u=zuul n=ansible | included: /home/zuul/src/github.com/openstack-k8s-operators/telemetry-operator/ci/default-telemetry/tasks/no_coo_no_autoscaling_no_metricstorage.yml for localhost 2026-02-02 11:23:55,244 p=36171 u=zuul n=ansible | TASK [default-telemetry : Wait until telemetry is finished reconciling and its conditions are all True] *** 2026-02-02 11:23:55,244 p=36171 u=zuul n=ansible | Monday 02 February 2026 11:23:55 +0000 (0:00:00.034) 0:00:00.085 ******* 2026-02-02 11:23:55,789 p=36171 u=zuul n=ansible | changed: [localhost] 2026-02-02 11:23:55,793 p=36171 u=zuul n=ansible | TASK [default-telemetry : Get current time] ************************************ 2026-02-02 11:23:55,793 p=36171 u=zuul n=ansible | Monday 02 February 2026 11:23:55 +0000 (0:00:00.548) 0:00:00.634 ******* 2026-02-02 11:23:55,809 p=36171 u=zuul n=ansible | ok: [localhost] 2026-02-02 11:23:55,813 p=36171 u=zuul n=ansible | TASK [default-telemetry : Wait for new logs to be generated] ******************* 2026-02-02 11:23:55,813 p=36171 u=zuul n=ansible | Monday 02 February 2026 11:23:55 +0000 (0:00:00.020) 0:00:00.654 ******* 2026-02-02 11:25:56,237 p=36171 u=zuul n=ansible | ok: [localhost] 2026-02-02 11:25:56,242 p=36171 u=zuul n=ansible | TASK [default-telemetry : Get new telemetry-operator logs] ********************* 2026-02-02 11:25:56,242 p=36171 u=zuul n=ansible | Monday 02 February 2026 11:25:56 +0000 (0:02:00.429) 0:02:01.083 ******* 2026-02-02 11:25:56,590 p=36171 u=zuul n=ansible | changed: [localhost] 2026-02-02 11:25:56,594 p=36171 u=zuul n=ansible | TASK [default-telemetry : TEST Check telemetry-operator logs for errors after it's started RHOSO-123456] *** 2026-02-02 11:25:56,594 p=36171 u=zuul n=ansible | Monday 02 February 2026 11:25:56 +0000 (0:00:00.351) 0:02:01.435 ******* 2026-02-02 11:25:56,616 p=36171 u=zuul n=ansible | ok: [localhost] 2026-02-02 11:25:56,621 p=36171 u=zuul n=ansible | TASK [default-telemetry : Output found errors for debugging purposes] ********** 2026-02-02 11:25:56,621 p=36171 u=zuul n=ansible | Monday 02 February 2026 11:25:56 +0000 (0:00:00.027) 0:02:01.462 ******* 2026-02-02 11:25:56,633 p=36171 u=zuul n=ansible | skipping: [localhost] 2026-02-02 11:25:56,637 p=36171 u=zuul n=ansible | TASK [default-telemetry : Get telemetry-operator pod data] ********************* 2026-02-02 11:25:56,637 p=36171 u=zuul n=ansible | Monday 02 February 2026 11:25:56 +0000 (0:00:00.015) 0:02:01.478 ******* 2026-02-02 11:25:56,997 p=36171 u=zuul n=ansible | changed: [localhost] 2026-02-02 11:25:57,001 p=36171 u=zuul n=ansible | TASK [default-telemetry : Get telemetry-operator container statuses] *********** 2026-02-02 11:25:57,002 p=36171 u=zuul n=ansible | Monday 02 February 2026 11:25:57 +0000 (0:00:00.364) 0:02:01.843 ******* 2026-02-02 11:25:57,018 p=36171 u=zuul n=ansible | ok: [localhost] 2026-02-02 11:25:57,022 p=36171 u=zuul n=ansible | TASK [default-telemetry : Get telemetry-operator restart counts] *************** 2026-02-02 11:25:57,022 p=36171 u=zuul n=ansible | Monday 02 February 2026 11:25:57 +0000 (0:00:00.020) 0:02:01.863 ******* 2026-02-02 11:25:57,042 p=36171 u=zuul n=ansible | ok: [localhost] 2026-02-02 11:25:57,047 p=36171 u=zuul n=ansible | TASK [default-telemetry : Get telemetry-operator failed container logs] ******** 2026-02-02 11:25:57,047 p=36171 u=zuul n=ansible | Monday 02 February 2026 11:25:57 +0000 (0:00:00.025) 0:02:01.888 ******* 2026-02-02 11:25:57,427 p=36171 u=zuul n=ansible | fatal: [localhost]: FAILED! => {"changed": true, "cmd": ["oc", "logs", "-n", "openstack-operators", "-p", "-l", "openstack.org/operator-name=telemetry", "--tail=-1"], "delta": "0:00:00.178824", "end": "2026-02-02 11:25:57.401630", "msg": "non-zero return code", "rc": 1, "start": "2026-02-02 11:25:57.222806", "stderr": "Error from server (BadRequest): previous terminated container \"manager\" in pod \"telemetry-operator-controller-manager-84dbcd4d6-strlk\" not found", "stderr_lines": ["Error from server (BadRequest): previous terminated container \"manager\" in pod \"telemetry-operator-controller-manager-84dbcd4d6-strlk\" not found"], "stdout": "", "stdout_lines": []} 2026-02-02 11:25:57,427 p=36171 u=zuul n=ansible | ...ignoring 2026-02-02 11:25:57,431 p=36171 u=zuul n=ansible | TASK [default-telemetry : Output logs of failed container for debugging purposes] *** 2026-02-02 11:25:57,431 p=36171 u=zuul n=ansible | Monday 02 February 2026 11:25:57 +0000 (0:00:00.383) 0:02:02.272 ******* 2026-02-02 11:25:57,452 p=36171 u=zuul n=ansible | ok: [localhost] => { "operator_logs_previous.stdout_lines": [] } 2026-02-02 11:25:57,455 p=36171 u=zuul n=ansible | TASK [default-telemetry : Enable Autoscaling and MetricStorage] **************** 2026-02-02 11:25:57,455 p=36171 u=zuul n=ansible | Monday 02 February 2026 11:25:57 +0000 (0:00:00.024) 0:02:02.296 ******* 2026-02-02 11:25:57,471 p=36171 u=zuul n=ansible | included: /home/zuul/src/github.com/openstack-k8s-operators/telemetry-operator/ci/default-telemetry/tasks/enable_autoscaling_and_metricstorage.yml for localhost 2026-02-02 11:25:57,482 p=36171 u=zuul n=ansible | TASK [default-telemetry : Enable MetricStorage] ******************************** 2026-02-02 11:25:57,482 p=36171 u=zuul n=ansible | Monday 02 February 2026 11:25:57 +0000 (0:00:00.026) 0:02:02.323 ******* 2026-02-02 11:25:57,956 p=36171 u=zuul n=ansible | changed: [localhost] 2026-02-02 11:25:57,960 p=36171 u=zuul n=ansible | TASK [default-telemetry : Enable Autoscaling] ********************************** 2026-02-02 11:25:57,960 p=36171 u=zuul n=ansible | Monday 02 February 2026 11:25:57 +0000 (0:00:00.478) 0:02:02.801 ******* 2026-02-02 11:25:58,488 p=36171 u=zuul n=ansible | changed: [localhost] 2026-02-02 11:25:58,491 p=36171 u=zuul n=ansible | TASK [default-telemetry : Wait until reconciliation finishes] ****************** 2026-02-02 11:25:58,491 p=36171 u=zuul n=ansible | Monday 02 February 2026 11:25:58 +0000 (0:00:00.531) 0:02:03.332 ******* 2026-02-02 11:27:58,789 p=36171 u=zuul n=ansible | ok: [localhost] 2026-02-02 11:27:58,793 p=36171 u=zuul n=ansible | TASK [default-telemetry : Check telemetry without COO and with Autoscaling and MetricStorage enabled] *** 2026-02-02 11:27:58,793 p=36171 u=zuul n=ansible | Monday 02 February 2026 11:27:58 +0000 (0:02:00.301) 0:04:03.634 ******* 2026-02-02 11:27:58,833 p=36171 u=zuul n=ansible | included: /home/zuul/src/github.com/openstack-k8s-operators/telemetry-operator/ci/default-telemetry/tasks/no_coo_autoscaling_metricstorage.yml for localhost 2026-02-02 11:27:58,854 p=36171 u=zuul n=ansible | TASK [default-telemetry : Get telemetry-operator pod data] ********************* 2026-02-02 11:27:58,855 p=36171 u=zuul n=ansible | Monday 02 February 2026 11:27:58 +0000 (0:00:00.061) 0:04:03.695 ******* 2026-02-02 11:27:59,278 p=36171 u=zuul n=ansible | changed: [localhost] 2026-02-02 11:27:59,282 p=36171 u=zuul n=ansible | TASK [default-telemetry : Get telemetry-operator container statuses] *********** 2026-02-02 11:27:59,282 p=36171 u=zuul n=ansible | Monday 02 February 2026 11:27:59 +0000 (0:00:00.427) 0:04:04.123 ******* 2026-02-02 11:27:59,298 p=36171 u=zuul n=ansible | ok: [localhost] 2026-02-02 11:27:59,303 p=36171 u=zuul n=ansible | TASK [default-telemetry : Get telemetry-operator restart counts] *************** 2026-02-02 11:27:59,303 p=36171 u=zuul n=ansible | Monday 02 February 2026 11:27:59 +0000 (0:00:00.021) 0:04:04.144 ******* 2026-02-02 11:27:59,324 p=36171 u=zuul n=ansible | ok: [localhost] 2026-02-02 11:27:59,328 p=36171 u=zuul n=ansible | TASK [default-telemetry : Get telemetry-operator failed container logs] ******** 2026-02-02 11:27:59,328 p=36171 u=zuul n=ansible | Monday 02 February 2026 11:27:59 +0000 (0:00:00.024) 0:04:04.169 ******* 2026-02-02 11:27:59,684 p=36171 u=zuul n=ansible | fatal: [localhost]: FAILED! => {"changed": true, "cmd": ["oc", "logs", "-n", "openstack-operators", "-p", "-l", "openstack.org/operator-name=telemetry", "--tail=-1"], "delta": "0:00:00.151778", "end": "2026-02-02 11:27:59.657768", "msg": "non-zero return code", "rc": 1, "start": "2026-02-02 11:27:59.505990", "stderr": "Error from server (BadRequest): previous terminated container \"manager\" in pod \"telemetry-operator-controller-manager-84dbcd4d6-strlk\" not found", "stderr_lines": ["Error from server (BadRequest): previous terminated container \"manager\" in pod \"telemetry-operator-controller-manager-84dbcd4d6-strlk\" not found"], "stdout": "", "stdout_lines": []} 2026-02-02 11:27:59,685 p=36171 u=zuul n=ansible | ...ignoring 2026-02-02 11:27:59,689 p=36171 u=zuul n=ansible | TASK [default-telemetry : Output logs of failed container for debugging purposes] *** 2026-02-02 11:27:59,690 p=36171 u=zuul n=ansible | Monday 02 February 2026 11:27:59 +0000 (0:00:00.361) 0:04:04.531 ******* 2026-02-02 11:27:59,706 p=36171 u=zuul n=ansible | ok: [localhost] => { "operator_logs_previous.stdout_lines": [] } 2026-02-02 11:27:59,711 p=36171 u=zuul n=ansible | TASK [default-telemetry : Get current time] ************************************ 2026-02-02 11:27:59,711 p=36171 u=zuul n=ansible | Monday 02 February 2026 11:27:59 +0000 (0:00:00.021) 0:04:04.552 ******* 2026-02-02 11:27:59,728 p=36171 u=zuul n=ansible | ok: [localhost] 2026-02-02 11:27:59,732 p=36171 u=zuul n=ansible | TASK [default-telemetry : Wait for new logs to be generated] ******************* 2026-02-02 11:27:59,732 p=36171 u=zuul n=ansible | Monday 02 February 2026 11:27:59 +0000 (0:00:00.021) 0:04:04.573 ******* 2026-02-02 11:29:59,982 p=36171 u=zuul n=ansible | ok: [localhost] 2026-02-02 11:29:59,986 p=36171 u=zuul n=ansible | TASK [default-telemetry : Get new telemetry-operator logs] ********************* 2026-02-02 11:29:59,986 p=36171 u=zuul n=ansible | Monday 02 February 2026 11:29:59 +0000 (0:02:00.253) 0:06:04.827 ******* 2026-02-02 11:30:00,360 p=36171 u=zuul n=ansible | changed: [localhost] 2026-02-02 11:30:00,367 p=36171 u=zuul n=ansible | TASK [default-telemetry : TEST Check that telemetry-operator logs include expected error logs when MetricStorage is enabled, but COO isn't installed RHOSO-123459] *** 2026-02-02 11:30:00,367 p=36171 u=zuul n=ansible | Monday 02 February 2026 11:30:00 +0000 (0:00:00.381) 0:06:05.208 ******* 2026-02-02 11:30:00,387 p=36171 u=zuul n=ansible | ok: [localhost] => (item=INFO Controllers.MetricStorage Can't own MonitoringStack resource) => { "ansible_loop_var": "item", "changed": false, "item": "INFO\tControllers.MetricStorage\tCan't own MonitoringStack resource", "msg": "All assertions passed" } 2026-02-02 11:30:00,394 p=36171 u=zuul n=ansible | TASK [default-telemetry : TEST Check that telemetry-operator logs don't include errors when MetricStorage is enabled, but COO isn't installed RHOSO-123460] *** 2026-02-02 11:30:00,394 p=36171 u=zuul n=ansible | Monday 02 February 2026 11:30:00 +0000 (0:00:00.026) 0:06:05.235 ******* 2026-02-02 11:30:00,410 p=36171 u=zuul n=ansible | fatal: [localhost]: FAILED! => {"ansible_facts": {"error_list": ["ERROR\tReconciler error\t{\"controller\": \"autoscaling\", \"controllerGroup\": \"telemetry.openstack.org\", \"controllerKind\": \"Autoscaling\", \"Autoscaling\": {\"name\":\"autoscaling\",\"namespace\":\"openstack\"}, \"namespace\": \"openstack\", \"name\": \"autoscaling\", \"reconcileID\": \"90f42b8c-ab99-4d11-a61f-0e0e2cd877d1\", \"error\": \"ApplicationCredential secret not found: ac-aodh-secret\"}"]}, "changed": false, "failed_when_result": true} 2026-02-02 11:30:00,411 p=36171 u=zuul n=ansible | ...ignoring 2026-02-02 11:30:00,414 p=36171 u=zuul n=ansible | TASK [default-telemetry : Get MetricStorage condition types] ******************* 2026-02-02 11:30:00,414 p=36171 u=zuul n=ansible | Monday 02 February 2026 11:30:00 +0000 (0:00:00.020) 0:06:05.255 ******* 2026-02-02 11:30:00,743 p=36171 u=zuul n=ansible | changed: [localhost] 2026-02-02 11:30:00,747 p=36171 u=zuul n=ansible | TASK [default-telemetry : Get MetricStorage condition values] ****************** 2026-02-02 11:30:00,747 p=36171 u=zuul n=ansible | Monday 02 February 2026 11:30:00 +0000 (0:00:00.333) 0:06:05.588 ******* 2026-02-02 11:30:01,116 p=36171 u=zuul n=ansible | changed: [localhost] => (item=Ready) 2026-02-02 11:30:01,489 p=36171 u=zuul n=ansible | changed: [localhost] => (item=DashboardDatasourceReady) 2026-02-02 11:30:01,823 p=36171 u=zuul n=ansible | changed: [localhost] => (item=DashboardDefinitionReady) 2026-02-02 11:30:02,161 p=36171 u=zuul n=ansible | changed: [localhost] => (item=DashboardPluginReady) 2026-02-02 11:30:02,497 p=36171 u=zuul n=ansible | changed: [localhost] => (item=DashboardPrometheusRuleReady) 2026-02-02 11:30:02,844 p=36171 u=zuul n=ansible | changed: [localhost] => (item=MonitoringStackReady) 2026-02-02 11:30:03,275 p=36171 u=zuul n=ansible | changed: [localhost] => (item=NetworkAttachmentsReady) 2026-02-02 11:30:03,723 p=36171 u=zuul n=ansible | changed: [localhost] => (item=PrometheusReady) 2026-02-02 11:30:04,107 p=36171 u=zuul n=ansible | changed: [localhost] => (item=ScrapeConfigReady) 2026-02-02 11:30:04,543 p=36171 u=zuul n=ansible | changed: [localhost] => (item=TLSInputReady) 2026-02-02 11:30:04,552 p=36171 u=zuul n=ansible | TASK [default-telemetry : Construct MetricStorage condition dictionary] ******** 2026-02-02 11:30:04,552 p=36171 u=zuul n=ansible | Monday 02 February 2026 11:30:04 +0000 (0:00:03.804) 0:06:09.393 ******* 2026-02-02 11:30:04,646 p=36171 u=zuul n=ansible | ok: [localhost] => (item={'changed': True, 'stdout': 'Error occured when trying to own: customresourcedefinitions.apiextensions.k8s.io "monitoringstacks.monitoring.rhobs" not found', 'stderr': '', 'rc': 0, 'cmd': ['oc', 'get', 'metricstorage', 'metric-storage', '-o', 'jsonpath={.status.conditions[?(@.type == "Ready")].message}'], 'start': '2026-02-02 11:30:00.934072', 'end': '2026-02-02 11:30:01.085751', 'delta': '0:00:00.151679', 'msg': '', 'invocation': {'module_args': {'_raw_params': 'oc get metricstorage metric-storage -o jsonpath=\'{.status.conditions[?(@.type == "Ready")].message}\'', '_uses_shell': False, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': ['Error occured when trying to own: customresourcedefinitions.apiextensions.k8s.io "monitoringstacks.monitoring.rhobs" not found'], 'stderr_lines': [], 'failed': False, 'item': 'Ready', 'ansible_loop_var': 'item'}) 2026-02-02 11:30:04,648 p=36171 u=zuul n=ansible | ok: [localhost] => (item={'changed': True, 'stdout': 'Dashboard Datasource not started', 'stderr': '', 'rc': 0, 'cmd': ['oc', 'get', 'metricstorage', 'metric-storage', '-o', 'jsonpath={.status.conditions[?(@.type == "DashboardDatasourceReady")].message}'], 'start': '2026-02-02 11:30:01.310112', 'end': '2026-02-02 11:30:01.463141', 'delta': '0:00:00.153029', 'msg': '', 'invocation': {'module_args': {'_raw_params': 'oc get metricstorage metric-storage -o jsonpath=\'{.status.conditions[?(@.type == "DashboardDatasourceReady")].message}\'', '_uses_shell': False, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': ['Dashboard Datasource not started'], 'stderr_lines': [], 'failed': False, 'item': 'DashboardDatasourceReady', 'ansible_loop_var': 'item'}) 2026-02-02 11:30:04,648 p=36171 u=zuul n=ansible | ok: [localhost] => (item={'changed': True, 'stdout': 'Dashboard Definition not started', 'stderr': '', 'rc': 0, 'cmd': ['oc', 'get', 'metricstorage', 'metric-storage', '-o', 'jsonpath={.status.conditions[?(@.type == "DashboardDefinitionReady")].message}'], 'start': '2026-02-02 11:30:01.637318', 'end': '2026-02-02 11:30:01.796131', 'delta': '0:00:00.158813', 'msg': '', 'invocation': {'module_args': {'_raw_params': 'oc get metricstorage metric-storage -o jsonpath=\'{.status.conditions[?(@.type == "DashboardDefinitionReady")].message}\'', '_uses_shell': False, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': ['Dashboard Definition not started'], 'stderr_lines': [], 'failed': False, 'item': 'DashboardDefinitionReady', 'ansible_loop_var': 'item'}) 2026-02-02 11:30:04,649 p=36171 u=zuul n=ansible | ok: [localhost] => (item={'changed': True, 'stdout': 'Dashboard Plugin not started', 'stderr': '', 'rc': 0, 'cmd': ['oc', 'get', 'metricstorage', 'metric-storage', '-o', 'jsonpath={.status.conditions[?(@.type == "DashboardPluginReady")].message}'], 'start': '2026-02-02 11:30:01.991837', 'end': '2026-02-02 11:30:02.128121', 'delta': '0:00:00.136284', 'msg': '', 'invocation': {'module_args': {'_raw_params': 'oc get metricstorage metric-storage -o jsonpath=\'{.status.conditions[?(@.type == "DashboardPluginReady")].message}\'', '_uses_shell': False, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': ['Dashboard Plugin not started'], 'stderr_lines': [], 'failed': False, 'item': 'DashboardPluginReady', 'ansible_loop_var': 'item'}) 2026-02-02 11:30:04,650 p=36171 u=zuul n=ansible | ok: [localhost] => (item={'changed': True, 'stdout': 'Dashboard PrometheusRule not started', 'stderr': '', 'rc': 0, 'cmd': ['oc', 'get', 'metricstorage', 'metric-storage', '-o', 'jsonpath={.status.conditions[?(@.type == "DashboardPrometheusRuleReady")].message}'], 'start': '2026-02-02 11:30:02.324286', 'end': '2026-02-02 11:30:02.470815', 'delta': '0:00:00.146529', 'msg': '', 'invocation': {'module_args': {'_raw_params': 'oc get metricstorage metric-storage -o jsonpath=\'{.status.conditions[?(@.type == "DashboardPrometheusRuleReady")].message}\'', '_uses_shell': False, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': ['Dashboard PrometheusRule not started'], 'stderr_lines': [], 'failed': False, 'item': 'DashboardPrometheusRuleReady', 'ansible_loop_var': 'item'}) 2026-02-02 11:30:04,661 p=36171 u=zuul n=ansible | ok: [localhost] => (item={'changed': True, 'stdout': 'Error occured when trying to own: customresourcedefinitions.apiextensions.k8s.io "monitoringstacks.monitoring.rhobs" not found', 'stderr': '', 'rc': 0, 'cmd': ['oc', 'get', 'metricstorage', 'metric-storage', '-o', 'jsonpath={.status.conditions[?(@.type == "MonitoringStackReady")].message}'], 'start': '2026-02-02 11:30:02.651441', 'end': '2026-02-02 11:30:02.808893', 'delta': '0:00:00.157452', 'msg': '', 'invocation': {'module_args': {'_raw_params': 'oc get metricstorage metric-storage -o jsonpath=\'{.status.conditions[?(@.type == "MonitoringStackReady")].message}\'', '_uses_shell': False, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': ['Error occured when trying to own: customresourcedefinitions.apiextensions.k8s.io "monitoringstacks.monitoring.rhobs" not found'], 'stderr_lines': [], 'failed': False, 'item': 'MonitoringStackReady', 'ansible_loop_var': 'item'}) 2026-02-02 11:30:04,669 p=36171 u=zuul n=ansible | ok: [localhost] => (item={'changed': True, 'stdout': 'NetworkAttachments not started', 'stderr': '', 'rc': 0, 'cmd': ['oc', 'get', 'metricstorage', 'metric-storage', '-o', 'jsonpath={.status.conditions[?(@.type == "NetworkAttachmentsReady")].message}'], 'start': '2026-02-02 11:30:03.034798', 'end': '2026-02-02 11:30:03.236065', 'delta': '0:00:00.201267', 'msg': '', 'invocation': {'module_args': {'_raw_params': 'oc get metricstorage metric-storage -o jsonpath=\'{.status.conditions[?(@.type == "NetworkAttachmentsReady")].message}\'', '_uses_shell': False, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': ['NetworkAttachments not started'], 'stderr_lines': [], 'failed': False, 'item': 'NetworkAttachmentsReady', 'ansible_loop_var': 'item'}) 2026-02-02 11:30:04,680 p=36171 u=zuul n=ansible | ok: [localhost] => (item={'changed': True, 'stdout': 'Prometheus not ready', 'stderr': '', 'rc': 0, 'cmd': ['oc', 'get', 'metricstorage', 'metric-storage', '-o', 'jsonpath={.status.conditions[?(@.type == "PrometheusReady")].message}'], 'start': '2026-02-02 11:30:03.486450', 'end': '2026-02-02 11:30:03.679223', 'delta': '0:00:00.192773', 'msg': '', 'invocation': {'module_args': {'_raw_params': 'oc get metricstorage metric-storage -o jsonpath=\'{.status.conditions[?(@.type == "PrometheusReady")].message}\'', '_uses_shell': False, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': ['Prometheus not ready'], 'stderr_lines': [], 'failed': False, 'item': 'PrometheusReady', 'ansible_loop_var': 'item'}) 2026-02-02 11:30:04,693 p=36171 u=zuul n=ansible | ok: [localhost] => (item={'changed': True, 'stdout': 'ScrapeConfig not started', 'stderr': '', 'rc': 0, 'cmd': ['oc', 'get', 'metricstorage', 'metric-storage', '-o', 'jsonpath={.status.conditions[?(@.type == "ScrapeConfigReady")].message}'], 'start': '2026-02-02 11:30:03.912287', 'end': '2026-02-02 11:30:04.076404', 'delta': '0:00:00.164117', 'msg': '', 'invocation': {'module_args': {'_raw_params': 'oc get metricstorage metric-storage -o jsonpath=\'{.status.conditions[?(@.type == "ScrapeConfigReady")].message}\'', '_uses_shell': False, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': ['ScrapeConfig not started'], 'stderr_lines': [], 'failed': False, 'item': 'ScrapeConfigReady', 'ansible_loop_var': 'item'}) 2026-02-02 11:30:04,700 p=36171 u=zuul n=ansible | ok: [localhost] => (item={'changed': True, 'stdout': 'Input data not checked', 'stderr': '', 'rc': 0, 'cmd': ['oc', 'get', 'metricstorage', 'metric-storage', '-o', 'jsonpath={.status.conditions[?(@.type == "TLSInputReady")].message}'], 'start': '2026-02-02 11:30:04.334238', 'end': '2026-02-02 11:30:04.513391', 'delta': '0:00:00.179153', 'msg': '', 'invocation': {'module_args': {'_raw_params': 'oc get metricstorage metric-storage -o jsonpath=\'{.status.conditions[?(@.type == "TLSInputReady")].message}\'', '_uses_shell': False, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': ['Input data not checked'], 'stderr_lines': [], 'failed': False, 'item': 'TLSInputReady', 'ansible_loop_var': 'item'}) 2026-02-02 11:30:04,708 p=36171 u=zuul n=ansible | TASK [default-telemetry : TEST Check that all MetricStorage conditions are as expected when COO isn't installed RHOSO-123461] *** 2026-02-02 11:30:04,708 p=36171 u=zuul n=ansible | Monday 02 February 2026 11:30:04 +0000 (0:00:00.156) 0:06:09.549 ******* 2026-02-02 11:30:04,740 p=36171 u=zuul n=ansible | ok: [localhost] => (item={'key': 'MonitoringStackReady', 'value': 'Error occured when trying to own: customresourcedefinitions.apiextensions.k8s.io "monitoringstacks.monitoring.rhobs" not found'}) => { "ansible_loop_var": "item", "changed": false, "item": { "key": "MonitoringStackReady", "value": "Error occured when trying to own: customresourcedefinitions.apiextensions.k8s.io \"monitoringstacks.monitoring.rhobs\" not found" }, "msg": "All assertions passed" } 2026-02-02 11:30:04,743 p=36171 u=zuul n=ansible | ok: [localhost] => (item={'key': 'Ready', 'value': 'Error occured when trying to own: customresourcedefinitions.apiextensions.k8s.io "monitoringstacks.monitoring.rhobs" not found'}) => { "ansible_loop_var": "item", "changed": false, "item": { "key": "Ready", "value": "Error occured when trying to own: customresourcedefinitions.apiextensions.k8s.io \"monitoringstacks.monitoring.rhobs\" not found" }, "msg": "All assertions passed" } 2026-02-02 11:30:04,748 p=36171 u=zuul n=ansible | TASK [default-telemetry : Get Autoscaling condition types] ********************* 2026-02-02 11:30:04,748 p=36171 u=zuul n=ansible | Monday 02 February 2026 11:30:04 +0000 (0:00:00.040) 0:06:09.589 ******* 2026-02-02 11:30:05,139 p=36171 u=zuul n=ansible | changed: [localhost] 2026-02-02 11:30:05,143 p=36171 u=zuul n=ansible | TASK [default-telemetry : Get Autoscaling condition values] ******************** 2026-02-02 11:30:05,143 p=36171 u=zuul n=ansible | Monday 02 February 2026 11:30:05 +0000 (0:00:00.394) 0:06:09.984 ******* 2026-02-02 11:30:05,564 p=36171 u=zuul n=ansible | changed: [localhost] => (item=Ready) 2026-02-02 11:30:05,950 p=36171 u=zuul n=ansible | changed: [localhost] => (item=DBReady) 2026-02-02 11:30:06,336 p=36171 u=zuul n=ansible | changed: [localhost] => (item=DBSyncReady) 2026-02-02 11:30:06,685 p=36171 u=zuul n=ansible | changed: [localhost] => (item=DeploymentReady) 2026-02-02 11:30:06,978 p=36171 u=zuul n=ansible | changed: [localhost] => (item=HeatReady) 2026-02-02 11:30:07,318 p=36171 u=zuul n=ansible | changed: [localhost] => (item=KeystoneEndpointReady) 2026-02-02 11:30:07,656 p=36171 u=zuul n=ansible | changed: [localhost] => (item=KeystoneServiceReady) 2026-02-02 11:30:07,990 p=36171 u=zuul n=ansible | changed: [localhost] => (item=MariaDBAccountReady) 2026-02-02 11:30:08,343 p=36171 u=zuul n=ansible | changed: [localhost] => (item=MemcachedReady) 2026-02-02 11:30:08,693 p=36171 u=zuul n=ansible | changed: [localhost] => (item=RabbitMqTransportURLReady) 2026-02-02 11:30:09,029 p=36171 u=zuul n=ansible | changed: [localhost] => (item=RoleBindingReady) 2026-02-02 11:30:09,396 p=36171 u=zuul n=ansible | changed: [localhost] => (item=RoleReady) 2026-02-02 11:30:09,721 p=36171 u=zuul n=ansible | changed: [localhost] => (item=ServiceAccountReady) 2026-02-02 11:30:10,096 p=36171 u=zuul n=ansible | changed: [localhost] => (item=ServiceConfigReady) 2026-02-02 11:30:10,433 p=36171 u=zuul n=ansible | changed: [localhost] => (item=TLSInputReady) 2026-02-02 11:30:10,443 p=36171 u=zuul n=ansible | TASK [default-telemetry : Construct Autoscaling condition dictionary] ********** 2026-02-02 11:30:10,444 p=36171 u=zuul n=ansible | Monday 02 February 2026 11:30:10 +0000 (0:00:05.300) 0:06:15.284 ******* 2026-02-02 11:30:10,479 p=36171 u=zuul n=ansible | ok: [localhost] => (item={'changed': True, 'stdout': 'Service config create error occurred ApplicationCredential secret not found: ac-aodh-secret', 'stderr': '', 'rc': 0, 'cmd': ['oc', 'get', 'autoscaling', 'autoscaling', '-o', 'jsonpath={.status.conditions[?(@.type == "Ready")].message}'], 'start': '2026-02-02 11:30:05.358377', 'end': '2026-02-02 11:30:05.525275', 'delta': '0:00:00.166898', 'msg': '', 'invocation': {'module_args': {'_raw_params': 'oc get autoscaling autoscaling -o jsonpath=\'{.status.conditions[?(@.type == "Ready")].message}\'', '_uses_shell': False, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': ['Service config create error occurred ApplicationCredential secret not found: ac-aodh-secret'], 'stderr_lines': [], 'failed': False, 'item': 'Ready', 'ansible_loop_var': 'item'}) 2026-02-02 11:30:10,489 p=36171 u=zuul n=ansible | ok: [localhost] => (item={'changed': True, 'stdout': 'DB create completed', 'stderr': '', 'rc': 0, 'cmd': ['oc', 'get', 'autoscaling', 'autoscaling', '-o', 'jsonpath={.status.conditions[?(@.type == "DBReady")].message}'], 'start': '2026-02-02 11:30:05.762213', 'end': '2026-02-02 11:30:05.913246', 'delta': '0:00:00.151033', 'msg': '', 'invocation': {'module_args': {'_raw_params': 'oc get autoscaling autoscaling -o jsonpath=\'{.status.conditions[?(@.type == "DBReady")].message}\'', '_uses_shell': False, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': ['DB create completed'], 'stderr_lines': [], 'failed': False, 'item': 'DBReady', 'ansible_loop_var': 'item'}) 2026-02-02 11:30:10,498 p=36171 u=zuul n=ansible | ok: [localhost] => (item={'changed': True, 'stdout': 'DBsync not started', 'stderr': '', 'rc': 0, 'cmd': ['oc', 'get', 'autoscaling', 'autoscaling', '-o', 'jsonpath={.status.conditions[?(@.type == "DBSyncReady")].message}'], 'start': '2026-02-02 11:30:06.142695', 'end': '2026-02-02 11:30:06.297539', 'delta': '0:00:00.154844', 'msg': '', 'invocation': {'module_args': {'_raw_params': 'oc get autoscaling autoscaling -o jsonpath=\'{.status.conditions[?(@.type == "DBSyncReady")].message}\'', '_uses_shell': False, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': ['DBsync not started'], 'stderr_lines': [], 'failed': False, 'item': 'DBSyncReady', 'ansible_loop_var': 'item'}) 2026-02-02 11:30:10,509 p=36171 u=zuul n=ansible | ok: [localhost] => (item={'changed': True, 'stdout': 'Deployment not started', 'stderr': '', 'rc': 0, 'cmd': ['oc', 'get', 'autoscaling', 'autoscaling', '-o', 'jsonpath={.status.conditions[?(@.type == "DeploymentReady")].message}'], 'start': '2026-02-02 11:30:06.513199', 'end': '2026-02-02 11:30:06.660442', 'delta': '0:00:00.147243', 'msg': '', 'invocation': {'module_args': {'_raw_params': 'oc get autoscaling autoscaling -o jsonpath=\'{.status.conditions[?(@.type == "DeploymentReady")].message}\'', '_uses_shell': False, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': ['Deployment not started'], 'stderr_lines': [], 'failed': False, 'item': 'DeploymentReady', 'ansible_loop_var': 'item'}) 2026-02-02 11:30:10,519 p=36171 u=zuul n=ansible | ok: [localhost] => (item={'changed': True, 'stdout': 'Setup complete', 'stderr': '', 'rc': 0, 'cmd': ['oc', 'get', 'autoscaling', 'autoscaling', '-o', 'jsonpath={.status.conditions[?(@.type == "HeatReady")].message}'], 'start': '2026-02-02 11:30:06.824164', 'end': '2026-02-02 11:30:06.951479', 'delta': '0:00:00.127315', 'msg': '', 'invocation': {'module_args': {'_raw_params': 'oc get autoscaling autoscaling -o jsonpath=\'{.status.conditions[?(@.type == "HeatReady")].message}\'', '_uses_shell': False, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': ['Setup complete'], 'stderr_lines': [], 'failed': False, 'item': 'HeatReady', 'ansible_loop_var': 'item'}) 2026-02-02 11:30:10,526 p=36171 u=zuul n=ansible | ok: [localhost] => (item={'changed': True, 'stdout': '', 'stderr': '', 'rc': 0, 'cmd': ['oc', 'get', 'autoscaling', 'autoscaling', '-o', 'jsonpath={.status.conditions[?(@.type == "KeystoneEndpointReady")].message}'], 'start': '2026-02-02 11:30:07.129871', 'end': '2026-02-02 11:30:07.289770', 'delta': '0:00:00.159899', 'msg': '', 'invocation': {'module_args': {'_raw_params': 'oc get autoscaling autoscaling -o jsonpath=\'{.status.conditions[?(@.type == "KeystoneEndpointReady")].message}\'', '_uses_shell': False, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': [], 'stderr_lines': [], 'failed': False, 'item': 'KeystoneEndpointReady', 'ansible_loop_var': 'item'}) 2026-02-02 11:30:10,534 p=36171 u=zuul n=ansible | ok: [localhost] => (item={'changed': True, 'stdout': '', 'stderr': '', 'rc': 0, 'cmd': ['oc', 'get', 'autoscaling', 'autoscaling', '-o', 'jsonpath={.status.conditions[?(@.type == "KeystoneServiceReady")].message}'], 'start': '2026-02-02 11:30:07.477169', 'end': '2026-02-02 11:30:07.624148', 'delta': '0:00:00.146979', 'msg': '', 'invocation': {'module_args': {'_raw_params': 'oc get autoscaling autoscaling -o jsonpath=\'{.status.conditions[?(@.type == "KeystoneServiceReady")].message}\'', '_uses_shell': False, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': [], 'stderr_lines': [], 'failed': False, 'item': 'KeystoneServiceReady', 'ansible_loop_var': 'item'}) 2026-02-02 11:30:10,542 p=36171 u=zuul n=ansible | ok: [localhost] => (item={'changed': True, 'stdout': 'MariaDBAccount creation complete', 'stderr': '', 'rc': 0, 'cmd': ['oc', 'get', 'autoscaling', 'autoscaling', '-o', 'jsonpath={.status.conditions[?(@.type == "MariaDBAccountReady")].message}'], 'start': '2026-02-02 11:30:07.809295', 'end': '2026-02-02 11:30:07.958556', 'delta': '0:00:00.149261', 'msg': '', 'invocation': {'module_args': {'_raw_params': 'oc get autoscaling autoscaling -o jsonpath=\'{.status.conditions[?(@.type == "MariaDBAccountReady")].message}\'', '_uses_shell': False, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': ['MariaDBAccount creation complete'], 'stderr_lines': [], 'failed': False, 'item': 'MariaDBAccountReady', 'ansible_loop_var': 'item'}) 2026-02-02 11:30:10,551 p=36171 u=zuul n=ansible | ok: [localhost] => (item={'changed': True, 'stdout': ' Memcached instance has been provisioned', 'stderr': '', 'rc': 0, 'cmd': ['oc', 'get', 'autoscaling', 'autoscaling', '-o', 'jsonpath={.status.conditions[?(@.type == "MemcachedReady")].message}'], 'start': '2026-02-02 11:30:08.167765', 'end': '2026-02-02 11:30:08.308106', 'delta': '0:00:00.140341', 'msg': '', 'invocation': {'module_args': {'_raw_params': 'oc get autoscaling autoscaling -o jsonpath=\'{.status.conditions[?(@.type == "MemcachedReady")].message}\'', '_uses_shell': False, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': [' Memcached instance has been provisioned'], 'stderr_lines': [], 'failed': False, 'item': 'MemcachedReady', 'ansible_loop_var': 'item'}) 2026-02-02 11:30:10,559 p=36171 u=zuul n=ansible | ok: [localhost] => (item={'changed': True, 'stdout': 'RabbitMqTransportURL successfully created', 'stderr': '', 'rc': 0, 'cmd': ['oc', 'get', 'autoscaling', 'autoscaling', '-o', 'jsonpath={.status.conditions[?(@.type == "RabbitMqTransportURLReady")].message}'], 'start': '2026-02-02 11:30:08.523321', 'end': '2026-02-02 11:30:08.662307', 'delta': '0:00:00.138986', 'msg': '', 'invocation': {'module_args': {'_raw_params': 'oc get autoscaling autoscaling -o jsonpath=\'{.status.conditions[?(@.type == "RabbitMqTransportURLReady")].message}\'', '_uses_shell': False, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': ['RabbitMqTransportURL successfully created'], 'stderr_lines': [], 'failed': False, 'item': 'RabbitMqTransportURLReady', 'ansible_loop_var': 'item'}) 2026-02-02 11:30:10,566 p=36171 u=zuul n=ansible | ok: [localhost] => (item={'changed': True, 'stdout': 'RoleBinding created', 'stderr': '', 'rc': 0, 'cmd': ['oc', 'get', 'autoscaling', 'autoscaling', '-o', 'jsonpath={.status.conditions[?(@.type == "RoleBindingReady")].message}'], 'start': '2026-02-02 11:30:08.848380', 'end': '2026-02-02 11:30:09.001273', 'delta': '0:00:00.152893', 'msg': '', 'invocation': {'module_args': {'_raw_params': 'oc get autoscaling autoscaling -o jsonpath=\'{.status.conditions[?(@.type == "RoleBindingReady")].message}\'', '_uses_shell': False, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': ['RoleBinding created'], 'stderr_lines': [], 'failed': False, 'item': 'RoleBindingReady', 'ansible_loop_var': 'item'}) 2026-02-02 11:30:10,576 p=36171 u=zuul n=ansible | ok: [localhost] => (item={'changed': True, 'stdout': 'Role created', 'stderr': '', 'rc': 0, 'cmd': ['oc', 'get', 'autoscaling', 'autoscaling', '-o', 'jsonpath={.status.conditions[?(@.type == "RoleReady")].message}'], 'start': '2026-02-02 11:30:09.193245', 'end': '2026-02-02 11:30:09.367325', 'delta': '0:00:00.174080', 'msg': '', 'invocation': {'module_args': {'_raw_params': 'oc get autoscaling autoscaling -o jsonpath=\'{.status.conditions[?(@.type == "RoleReady")].message}\'', '_uses_shell': False, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': ['Role created'], 'stderr_lines': [], 'failed': False, 'item': 'RoleReady', 'ansible_loop_var': 'item'}) 2026-02-02 11:30:10,583 p=36171 u=zuul n=ansible | ok: [localhost] => (item={'changed': True, 'stdout': 'ServiceAccount created', 'stderr': '', 'rc': 0, 'cmd': ['oc', 'get', 'autoscaling', 'autoscaling', '-o', 'jsonpath={.status.conditions[?(@.type == "ServiceAccountReady")].message}'], 'start': '2026-02-02 11:30:09.559434', 'end': '2026-02-02 11:30:09.696090', 'delta': '0:00:00.136656', 'msg': '', 'invocation': {'module_args': {'_raw_params': 'oc get autoscaling autoscaling -o jsonpath=\'{.status.conditions[?(@.type == "ServiceAccountReady")].message}\'', '_uses_shell': False, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': ['ServiceAccount created'], 'stderr_lines': [], 'failed': False, 'item': 'ServiceAccountReady', 'ansible_loop_var': 'item'}) 2026-02-02 11:30:10,589 p=36171 u=zuul n=ansible | ok: [localhost] => (item={'changed': True, 'stdout': 'Service config create error occurred ApplicationCredential secret not found: ac-aodh-secret', 'stderr': '', 'rc': 0, 'cmd': ['oc', 'get', 'autoscaling', 'autoscaling', '-o', 'jsonpath={.status.conditions[?(@.type == "ServiceConfigReady")].message}'], 'start': '2026-02-02 11:30:09.882821', 'end': '2026-02-02 11:30:10.070420', 'delta': '0:00:00.187599', 'msg': '', 'invocation': {'module_args': {'_raw_params': 'oc get autoscaling autoscaling -o jsonpath=\'{.status.conditions[?(@.type == "ServiceConfigReady")].message}\'', '_uses_shell': False, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': ['Service config create error occurred ApplicationCredential secret not found: ac-aodh-secret'], 'stderr_lines': [], 'failed': False, 'item': 'ServiceConfigReady', 'ansible_loop_var': 'item'}) 2026-02-02 11:30:10,594 p=36171 u=zuul n=ansible | ok: [localhost] => (item={'changed': True, 'stdout': 'Input data not checked', 'stderr': '', 'rc': 0, 'cmd': ['oc', 'get', 'autoscaling', 'autoscaling', '-o', 'jsonpath={.status.conditions[?(@.type == "TLSInputReady")].message}'], 'start': '2026-02-02 11:30:10.255923', 'end': '2026-02-02 11:30:10.407996', 'delta': '0:00:00.152073', 'msg': '', 'invocation': {'module_args': {'_raw_params': 'oc get autoscaling autoscaling -o jsonpath=\'{.status.conditions[?(@.type == "TLSInputReady")].message}\'', '_uses_shell': False, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': ['Input data not checked'], 'stderr_lines': [], 'failed': False, 'item': 'TLSInputReady', 'ansible_loop_var': 'item'}) 2026-02-02 11:30:10,609 p=36171 u=zuul n=ansible | TASK [default-telemetry : TEST Check that all Autoscaling conditions are as expected when COO isn't installed RHOSO-123462] *** 2026-02-02 11:30:10,609 p=36171 u=zuul n=ansible | Monday 02 February 2026 11:30:10 +0000 (0:00:00.165) 0:06:15.450 ******* 2026-02-02 11:30:10,634 p=36171 u=zuul n=ansible | failed: [localhost] (item={'key': 'Ready', 'value': 'Setup complete'}) => { "ansible_loop_var": "item", "assertion": "conditions[item.key] == item.value", "changed": false, "evaluated_to": false, "item": { "key": "Ready", "value": "Setup complete" }, "msg": "Expected Ready condition field to be Setup complete, not {'results': [{'changed': True, 'stdout': 'Service config create error occurred ApplicationCredential secret not found: ac-aodh-secret', 'stderr': '', 'rc': 0, 'cmd': ['oc', 'get', 'autoscaling', 'autoscaling', '-o', 'jsonpath={.status.conditions[?(@.type == \"Ready\")].message}'], 'start': '2026-02-02 11:30:05.358377', 'end': '2026-02-02 11:30:05.525275', 'delta': '0:00:00.166898', 'msg': '', 'invocation': {'module_args': {'_raw_params': 'oc get autoscaling autoscaling -o jsonpath=\\'{.status.conditions[?(@.type == \"Ready\")].message}\\'', '_uses_shell': False, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': ['Service config create error occurred ApplicationCredential secret not found: ac-aodh-secret'], 'stderr_lines': [], 'failed': False, 'item': 'Ready', 'ansible_loop_var': 'item'}, {'changed': True, 'stdout': 'DB create completed', 'stderr': '', 'rc': 0, 'cmd': ['oc', 'get', 'autoscaling', 'autoscaling', '-o', 'jsonpath={.status.conditions[?(@.type == \"DBReady\")].message}'], 'start': '2026-02-02 11:30:05.762213', 'end': '2026-02-02 11:30:05.913246', 'delta': '0:00:00.151033', 'msg': '', 'invocation': {'module_args': {'_raw_params': 'oc get autoscaling autoscaling -o jsonpath=\\'{.status.conditions[?(@.type == \"DBReady\")].message}\\'', '_uses_shell': False, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': ['DB create completed'], 'stderr_lines': [], 'failed': False, 'item': 'DBReady', 'ansible_loop_var': 'item'}, {'changed': True, 'stdout': 'DBsync not started', 'stderr': '', 'rc': 0, 'cmd': ['oc', 'get', 'autoscaling', 'autoscaling', '-o', 'jsonpath={.status.conditions[?(@.type == \"DBSyncReady\")].message}'], 'start': '2026-02-02 11:30:06.142695', 'end': '2026-02-02 11:30:06.297539', 'delta': '0:00:00.154844', 'msg': '', 'invocation': {'module_args': {'_raw_params': 'oc get autoscaling autoscaling -o jsonpath=\\'{.status.conditions[?(@.type == \"DBSyncReady\")].message}\\'', '_uses_shell': False, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': ['DBsync not started'], 'stderr_lines': [], 'failed': False, 'item': 'DBSyncReady', 'ansible_loop_var': 'item'}, {'changed': True, 'stdout': 'Deployment not started', 'stderr': '', 'rc': 0, 'cmd': ['oc', 'get', 'autoscaling', 'autoscaling', '-o', 'jsonpath={.status.conditions[?(@.type == \"DeploymentReady\")].message}'], 'start': '2026-02-02 11:30:06.513199', 'end': '2026-02-02 11:30:06.660442', 'delta': '0:00:00.147243', 'msg': '', 'invocation': {'module_args': {'_raw_params': 'oc get autoscaling autoscaling -o jsonpath=\\'{.status.conditions[?(@.type == \"DeploymentReady\")].message}\\'', '_uses_shell': False, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': ['Deployment not started'], 'stderr_lines': [], 'failed': False, 'item': 'DeploymentReady', 'ansible_loop_var': 'item'}, {'changed': True, 'stdout': 'Setup complete', 'stderr': '', 'rc': 0, 'cmd': ['oc', 'get', 'autoscaling', 'autoscaling', '-o', 'jsonpath={.status.conditions[?(@.type == \"HeatReady\")].message}'], 'start': '2026-02-02 11:30:06.824164', 'end': '2026-02-02 11:30:06.951479', 'delta': '0:00:00.127315', 'msg': '', 'invocation': {'module_args': {'_raw_params': 'oc get autoscaling autoscaling -o jsonpath=\\'{.status.conditions[?(@.type == \"HeatReady\")].message}\\'', '_uses_shell': False, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': ['Setup complete'], 'stderr_lines': [], 'failed': False, 'item': 'HeatReady', 'ansible_loop_var': 'item'}, {'changed': True, 'stdout': '', 'stderr': '', 'rc': 0, 'cmd': ['oc', 'get', 'autoscaling', 'autoscaling', '-o', 'jsonpath={.status.conditions[?(@.type == \"KeystoneEndpointReady\")].message}'], 'start': '2026-02-02 11:30:07.129871', 'end': '2026-02-02 11:30:07.289770', 'delta': '0:00:00.159899', 'msg': '', 'invocation': {'module_args': {'_raw_params': 'oc get autoscaling autoscaling -o jsonpath=\\'{.status.conditions[?(@.type == \"KeystoneEndpointReady\")].message}\\'', '_uses_shell': False, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': [], 'stderr_lines': [], 'failed': False, 'item': 'KeystoneEndpointReady', 'ansible_loop_var': 'item'}, {'changed': True, 'stdout': '', 'stderr': '', 'rc': 0, 'cmd': ['oc', 'get', 'autoscaling', 'autoscaling', '-o', 'jsonpath={.status.conditions[?(@.type == \"KeystoneServiceReady\")].message}'], 'start': '2026-02-02 11:30:07.477169', 'end': '2026-02-02 11:30:07.624148', 'delta': '0:00:00.146979', 'msg': '', 'invocation': {'module_args': {'_raw_params': 'oc get autoscaling autoscaling -o jsonpath=\\'{.status.conditions[?(@.type == \"KeystoneServiceReady\")].message}\\'', '_uses_shell': False, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': [], 'stderr_lines': [], 'failed': False, 'item': 'KeystoneServiceReady', 'ansible_loop_var': 'item'}, {'changed': True, 'stdout': 'MariaDBAccount creation complete', 'stderr': '', 'rc': 0, 'cmd': ['oc', 'get', 'autoscaling', 'autoscaling', '-o', 'jsonpath={.status.conditions[?(@.type == \"MariaDBAccountReady\")].message}'], 'start': '2026-02-02 11:30:07.809295', 'end': '2026-02-02 11:30:07.958556', 'delta': '0:00:00.149261', 'msg': '', 'invocation': {'module_args': {'_raw_params': 'oc get autoscaling autoscaling -o jsonpath=\\'{.status.conditions[?(@.type == \"MariaDBAccountReady\")].message}\\'', '_uses_shell': False, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': ['MariaDBAccount creation complete'], 'stderr_lines': [], 'failed': False, 'item': 'MariaDBAccountReady', 'ansible_loop_var': 'item'}, {'changed': True, 'stdout': ' Memcached instance has been provisioned', 'stderr': '', 'rc': 0, 'cmd': ['oc', 'get', 'autoscaling', 'autoscaling', '-o', 'jsonpath={.status.conditions[?(@.type == \"MemcachedReady\")].message}'], 'start': '2026-02-02 11:30:08.167765', 'end': '2026-02-02 11:30:08.308106', 'delta': '0:00:00.140341', 'msg': '', 'invocation': {'module_args': {'_raw_params': 'oc get autoscaling autoscaling -o jsonpath=\\'{.status.conditions[?(@.type == \"MemcachedReady\")].message}\\'', '_uses_shell': False, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': [' Memcached instance has been provisioned'], 'stderr_lines': [], 'failed': False, 'item': 'MemcachedReady', 'ansible_loop_var': 'item'}, {'changed': True, 'stdout': 'RabbitMqTransportURL successfully created', 'stderr': '', 'rc': 0, 'cmd': ['oc', 'get', 'autoscaling', 'autoscaling', '-o', 'jsonpath={.status.conditions[?(@.type == \"RabbitMqTransportURLReady\")].message}'], 'start': '2026-02-02 11:30:08.523321', 'end': '2026-02-02 11:30:08.662307', 'delta': '0:00:00.138986', 'msg': '', 'invocation': {'module_args': {'_raw_params': 'oc get autoscaling autoscaling -o jsonpath=\\'{.status.conditions[?(@.type == \"RabbitMqTransportURLReady\")].message}\\'', '_uses_shell': False, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': ['RabbitMqTransportURL successfully created'], 'stderr_lines': [], 'failed': False, 'item': 'RabbitMqTransportURLReady', 'ansible_loop_var': 'item'}, {'changed': True, 'stdout': 'RoleBinding created', 'stderr': '', 'rc': 0, 'cmd': ['oc', 'get', 'autoscaling', 'autoscaling', '-o', 'jsonpath={.status.conditions[?(@.type == \"RoleBindingReady\")].message}'], 'start': '2026-02-02 11:30:08.848380', 'end': '2026-02-02 11:30:09.001273', 'delta': '0:00:00.152893', 'msg': '', 'invocation': {'module_args': {'_raw_params': 'oc get autoscaling autoscaling -o jsonpath=\\'{.status.conditions[?(@.type == \"RoleBindingReady\")].message}\\'', '_uses_shell': False, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': ['RoleBinding created'], 'stderr_lines': [], 'failed': False, 'item': 'RoleBindingReady', 'ansible_loop_var': 'item'}, {'changed': True, 'stdout': 'Role created', 'stderr': '', 'rc': 0, 'cmd': ['oc', 'get', 'autoscaling', 'autoscaling', '-o', 'jsonpath={.status.conditions[?(@.type == \"RoleReady\")].message}'], 'start': '2026-02-02 11:30:09.193245', 'end': '2026-02-02 11:30:09.367325', 'delta': '0:00:00.174080', 'msg': '', 'invocation': {'module_args': {'_raw_params': 'oc get autoscaling autoscaling -o jsonpath=\\'{.status.conditions[?(@.type == \"RoleReady\")].message}\\'', '_uses_shell': False, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': ['Role created'], 'stderr_lines': [], 'failed': False, 'item': 'RoleReady', 'ansible_loop_var': 'item'}, {'changed': True, 'stdout': 'ServiceAccount created', 'stderr': '', 'rc': 0, 'cmd': ['oc', 'get', 'autoscaling', 'autoscaling', '-o', 'jsonpath={.status.conditions[?(@.type == \"ServiceAccountReady\")].message}'], 'start': '2026-02-02 11:30:09.559434', 'end': '2026-02-02 11:30:09.696090', 'delta': '0:00:00.136656', 'msg': '', 'invocation': {'module_args': {'_raw_params': 'oc get autoscaling autoscaling -o jsonpath=\\'{.status.conditions[?(@.type == \"ServiceAccountReady\")].message}\\'', '_uses_shell': False, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': ['ServiceAccount created'], 'stderr_lines': [], 'failed': False, 'item': 'ServiceAccountReady', 'ansible_loop_var': 'item'}, {'changed': True, 'stdout': 'Service config create error occurred ApplicationCredential secret not found: ac-aodh-secret', 'stderr': '', 'rc': 0, 'cmd': ['oc', 'get', 'autoscaling', 'autoscaling', '-o', 'jsonpath={.status.conditions[?(@.type == \"ServiceConfigReady\")].message}'], 'start': '2026-02-02 11:30:09.882821', 'end': '2026-02-02 11:30:10.070420', 'delta': '0:00:00.187599', 'msg': '', 'invocation': {'module_args': {'_raw_params': 'oc get autoscaling autoscaling -o jsonpath=\\'{.status.conditions[?(@.type == \"ServiceConfigReady\")].message}\\'', '_uses_shell': False, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': ['Service config create error occurred ApplicationCredential secret not found: ac-aodh-secret'], 'stderr_lines': [], 'failed': False, 'item': 'ServiceConfigReady', 'ansible_loop_var': 'item'}, {'changed': True, 'stdout': 'Input data not checked', 'stderr': '', 'rc': 0, 'cmd': ['oc', 'get', 'autoscaling', 'autoscaling', '-o', 'jsonpath={.status.conditions[?(@.type == \"TLSInputReady\")].message}'], 'start': '2026-02-02 11:30:10.255923', 'end': '2026-02-02 11:30:10.407996', 'delta': '0:00:00.152073', 'msg': '', 'invocation': {'module_args': {'_raw_params': 'oc get autoscaling autoscaling -o jsonpath=\\'{.status.conditions[?(@.type == \"TLSInputReady\")].message}\\'', '_uses_shell': False, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': ['Input data not checked'], 'stderr_lines': [], 'failed': False, 'item': 'TLSInputReady', 'ansible_loop_var': 'item'}], 'skipped': False, 'changed': True, 'msg': 'All items completed'}" } 2026-02-02 11:30:10,636 p=36171 u=zuul n=ansible | ...ignoring 2026-02-02 11:30:10,640 p=36171 u=zuul n=ansible | TASK [default-telemetry : Enable Autoscaling and MetricStorage] **************** 2026-02-02 11:30:10,640 p=36171 u=zuul n=ansible | Monday 02 February 2026 11:30:10 +0000 (0:00:00.031) 0:06:15.481 ******* 2026-02-02 11:30:10,654 p=36171 u=zuul n=ansible | included: /home/zuul/src/github.com/openstack-k8s-operators/telemetry-operator/ci/default-telemetry/tasks/../../create-coo-subscription.yaml for localhost 2026-02-02 11:30:10,677 p=36171 u=zuul n=ansible | TASK [default-telemetry : Create the COO subscription] ************************* 2026-02-02 11:30:10,677 p=36171 u=zuul n=ansible | Monday 02 February 2026 11:30:10 +0000 (0:00:00.037) 0:06:15.518 ******* 2026-02-02 11:30:11,025 p=36171 u=zuul n=ansible | changed: [localhost] 2026-02-02 11:30:11,029 p=36171 u=zuul n=ansible | TASK [default-telemetry : Wait for the required resource to be created] ******** 2026-02-02 11:30:11,029 p=36171 u=zuul n=ansible | Monday 02 February 2026 11:30:11 +0000 (0:00:00.351) 0:06:15.870 ******* 2026-02-02 11:30:32,075 p=36171 u=zuul n=ansible | changed: [localhost] 2026-02-02 11:30:32,079 p=36171 u=zuul n=ansible | TASK [default-telemetry : Wait for the resources to be available] ************** 2026-02-02 11:30:32,079 p=36171 u=zuul n=ansible | Monday 02 February 2026 11:30:32 +0000 (0:00:21.049) 0:06:36.920 ******* 2026-02-02 11:31:13,071 p=36171 u=zuul n=ansible | changed: [localhost] 2026-02-02 11:31:13,075 p=36171 u=zuul n=ansible | TASK [default-telemetry : Check telemetry with COO and with Autoscaling and MetricStorage enabled] *** 2026-02-02 11:31:13,075 p=36171 u=zuul n=ansible | Monday 02 February 2026 11:31:13 +0000 (0:00:40.996) 0:07:17.916 ******* 2026-02-02 11:31:13,092 p=36171 u=zuul n=ansible | included: /home/zuul/src/github.com/openstack-k8s-operators/telemetry-operator/ci/default-telemetry/tasks/coo_autoscaling_metricstorage.yml for localhost 2026-02-02 11:31:13,116 p=36171 u=zuul n=ansible | TASK [default-telemetry : Wait until Autoscaling and MetricStorage are ready] *** 2026-02-02 11:31:13,116 p=36171 u=zuul n=ansible | Monday 02 February 2026 11:31:13 +0000 (0:00:00.041) 0:07:17.957 ******* 2026-02-02 11:32:26,118 p=36171 u=zuul n=ansible | changed: [localhost] 2026-02-02 11:32:26,122 p=36171 u=zuul n=ansible | TASK [default-telemetry : Get current time] ************************************ 2026-02-02 11:32:26,122 p=36171 u=zuul n=ansible | Monday 02 February 2026 11:32:26 +0000 (0:01:13.005) 0:08:30.963 ******* 2026-02-02 11:32:26,139 p=36171 u=zuul n=ansible | ok: [localhost] 2026-02-02 11:32:26,143 p=36171 u=zuul n=ansible | TASK [default-telemetry : Wait for new logs to be generated] ******************* 2026-02-02 11:32:26,144 p=36171 u=zuul n=ansible | Monday 02 February 2026 11:32:26 +0000 (0:00:00.021) 0:08:30.985 ******* 2026-02-02 11:34:26,365 p=36171 u=zuul n=ansible | ok: [localhost] 2026-02-02 11:34:26,369 p=36171 u=zuul n=ansible | TASK [default-telemetry : Get new telemetry-operator logs] ********************* 2026-02-02 11:34:26,369 p=36171 u=zuul n=ansible | Monday 02 February 2026 11:34:26 +0000 (0:02:00.225) 0:10:31.210 ******* 2026-02-02 11:34:26,762 p=36171 u=zuul n=ansible | changed: [localhost] 2026-02-02 11:34:26,768 p=36171 u=zuul n=ansible | TASK [default-telemetry : TEST Check that telemetry-operator logs don't include any errors after COO is installed RHOSO-123463] *** 2026-02-02 11:34:26,768 p=36171 u=zuul n=ansible | Monday 02 February 2026 11:34:26 +0000 (0:00:00.398) 0:10:31.609 ******* 2026-02-02 11:34:26,790 p=36171 u=zuul n=ansible | ok: [localhost] => { "changed": false, "msg": "All assertions passed" } 2026-02-02 11:34:26,796 p=36171 u=zuul n=ansible | TASK [default-telemetry : Test CustomMonitoringStack] ************************** 2026-02-02 11:34:26,797 p=36171 u=zuul n=ansible | Monday 02 February 2026 11:34:26 +0000 (0:00:00.028) 0:10:31.638 ******* 2026-02-02 11:34:26,825 p=36171 u=zuul n=ansible | included: /home/zuul/src/github.com/openstack-k8s-operators/telemetry-operator/ci/default-telemetry/tasks/test_custom_monitoring_stack.yml for localhost 2026-02-02 11:34:26,854 p=36171 u=zuul n=ansible | TASK [default-telemetry : Patch MetricStorage to use CustomMonitoringStack field] *** 2026-02-02 11:34:26,855 p=36171 u=zuul n=ansible | Monday 02 February 2026 11:34:26 +0000 (0:00:00.058) 0:10:31.696 ******* 2026-02-02 11:34:27,370 p=36171 u=zuul n=ansible | changed: [localhost] 2026-02-02 11:34:27,374 p=36171 u=zuul n=ansible | TASK [default-telemetry : Wait until MetricStorage is ready] ******************* 2026-02-02 11:34:27,374 p=36171 u=zuul n=ansible | Monday 02 February 2026 11:34:27 +0000 (0:00:00.518) 0:10:32.215 ******* 2026-02-02 11:36:27,709 p=36171 u=zuul n=ansible | fatal: [localhost]: FAILED! => {"changed": true, "cmd": ["oc", "wait", "telemetry", "telemetry", "--for=condition=Ready", "--timeout=2m"], "delta": "0:02:00.125258", "end": "2026-02-02 11:36:27.684730", "msg": "non-zero return code", "rc": 1, "start": "2026-02-02 11:34:27.559472", "stderr": "error: timed out waiting for the condition on telemetries/telemetry", "stderr_lines": ["error: timed out waiting for the condition on telemetries/telemetry"], "stdout": "", "stdout_lines": []} 2026-02-02 11:36:27,709 p=36171 u=zuul n=ansible | ...ignoring 2026-02-02 11:36:27,713 p=36171 u=zuul n=ansible | TASK [default-telemetry : Get current time] ************************************ 2026-02-02 11:36:27,713 p=36171 u=zuul n=ansible | Monday 02 February 2026 11:36:27 +0000 (0:02:00.339) 0:12:32.554 ******* 2026-02-02 11:36:27,731 p=36171 u=zuul n=ansible | ok: [localhost] 2026-02-02 11:36:27,735 p=36171 u=zuul n=ansible | TASK [default-telemetry : Wait for new logs to be generated] ******************* 2026-02-02 11:36:27,735 p=36171 u=zuul n=ansible | Monday 02 February 2026 11:36:27 +0000 (0:00:00.021) 0:12:32.576 ******* 2026-02-02 11:38:28,046 p=36171 u=zuul n=ansible | ok: [localhost] 2026-02-02 11:38:28,049 p=36171 u=zuul n=ansible | TASK [default-telemetry : Get new telemetry-operator logs] ********************* 2026-02-02 11:38:28,049 p=36171 u=zuul n=ansible | Monday 02 February 2026 11:38:28 +0000 (0:02:00.314) 0:14:32.890 ******* 2026-02-02 11:38:28,426 p=36171 u=zuul n=ansible | changed: [localhost] 2026-02-02 11:38:28,430 p=36171 u=zuul n=ansible | TASK [default-telemetry : TEST Check that telemetry-operator logs don't include any errors when using the CustomMonitoringStack RHOSO-123464] *** 2026-02-02 11:38:28,431 p=36171 u=zuul n=ansible | Monday 02 February 2026 11:38:28 +0000 (0:00:00.381) 0:14:33.271 ******* 2026-02-02 11:38:28,447 p=36171 u=zuul n=ansible | ok: [localhost] 2026-02-02 11:38:28,450 p=36171 u=zuul n=ansible | TASK [default-telemetry : Patch Autoscaling to use a custom Prometheus instance] *** 2026-02-02 11:38:28,450 p=36171 u=zuul n=ansible | Monday 02 February 2026 11:38:28 +0000 (0:00:00.019) 0:14:33.291 ******* 2026-02-02 11:38:28,934 p=36171 u=zuul n=ansible | changed: [localhost] 2026-02-02 11:38:28,941 p=36171 u=zuul n=ansible | TASK [default-telemetry : Wait until Autoscaling is ready] ********************* 2026-02-02 11:38:28,941 p=36171 u=zuul n=ansible | Monday 02 February 2026 11:38:28 +0000 (0:00:00.491) 0:14:33.782 ******* 2026-02-02 11:40:29,282 p=36171 u=zuul n=ansible | fatal: [localhost]: FAILED! => {"changed": true, "cmd": ["oc", "wait", "telemetry", "telemetry", "--for=condition=Ready", "--timeout=2m"], "delta": "0:02:00.119679", "end": "2026-02-02 11:40:29.232598", "msg": "non-zero return code", "rc": 1, "start": "2026-02-02 11:38:29.112919", "stderr": "error: timed out waiting for the condition on telemetries/telemetry", "stderr_lines": ["error: timed out waiting for the condition on telemetries/telemetry"], "stdout": "", "stdout_lines": []} 2026-02-02 11:40:29,282 p=36171 u=zuul n=ansible | ...ignoring 2026-02-02 11:40:29,291 p=36171 u=zuul n=ansible | TASK [default-telemetry : Get current time] ************************************ 2026-02-02 11:40:29,291 p=36171 u=zuul n=ansible | Monday 02 February 2026 11:40:29 +0000 (0:02:00.349) 0:16:34.132 ******* 2026-02-02 11:40:29,321 p=36171 u=zuul n=ansible | ok: [localhost] 2026-02-02 11:40:29,327 p=36171 u=zuul n=ansible | TASK [default-telemetry : Wait for new logs to be generated] ******************* 2026-02-02 11:40:29,328 p=36171 u=zuul n=ansible | Monday 02 February 2026 11:40:29 +0000 (0:00:00.036) 0:16:34.169 ******* 2026-02-02 11:42:29,604 p=36171 u=zuul n=ansible | ok: [localhost] 2026-02-02 11:42:29,608 p=36171 u=zuul n=ansible | TASK [default-telemetry : Get new telemetry-operator logs] ********************* 2026-02-02 11:42:29,608 p=36171 u=zuul n=ansible | Monday 02 February 2026 11:42:29 +0000 (0:02:00.280) 0:18:34.449 ******* 2026-02-02 11:42:29,991 p=36171 u=zuul n=ansible | changed: [localhost] 2026-02-02 11:42:29,996 p=36171 u=zuul n=ansible | TASK [default-telemetry : TEST Check that telemetry-operator logs don't include any errors when using a custom Prometheus instance for Autoscaling RHOSO-123465] *** 2026-02-02 11:42:29,996 p=36171 u=zuul n=ansible | Monday 02 February 2026 11:42:29 +0000 (0:00:00.387) 0:18:34.837 ******* 2026-02-02 11:42:30,017 p=36171 u=zuul n=ansible | ok: [localhost] 2026-02-02 11:42:30,021 p=36171 u=zuul n=ansible | TASK [default-telemetry : TEST Check Prometheus host is set correctly in the aodh-evaluator container when using a custom Prometheus RHOSO-123466] *** 2026-02-02 11:42:30,021 p=36171 u=zuul n=ansible | Monday 02 February 2026 11:42:30 +0000 (0:00:00.025) 0:18:34.862 ******* 2026-02-02 11:42:30,347 p=36171 u=zuul n=ansible | fatal: [localhost]: FAILED! => {"changed": true, "cmd": "oc rsh -c aodh-evaluator aodh-0 cat /etc/openstack/prometheus.yaml | grep host | cut -d \" \" -f 2", "delta": "0:00:00.135482", "end": "2026-02-02 11:42:30.324127", "failed_when_result": true, "msg": "", "rc": 0, "start": "2026-02-02 11:42:30.188645", "stderr": "Error from server (NotFound): pods \"aodh-0\" not found", "stderr_lines": ["Error from server (NotFound): pods \"aodh-0\" not found"], "stdout": "", "stdout_lines": []} 2026-02-02 11:42:30,347 p=36171 u=zuul n=ansible | ...ignoring 2026-02-02 11:42:30,351 p=36171 u=zuul n=ansible | TASK [default-telemetry : TEST Check Prometheus port is set correctly in the aodh-evaluator container when using a custom Prometheus RHOSO-123467] *** 2026-02-02 11:42:30,351 p=36171 u=zuul n=ansible | Monday 02 February 2026 11:42:30 +0000 (0:00:00.329) 0:18:35.192 ******* 2026-02-02 11:42:30,666 p=36171 u=zuul n=ansible | fatal: [localhost]: FAILED! => {"changed": true, "cmd": "oc rsh -c aodh-evaluator aodh-0 cat /etc/openstack/prometheus.yaml | grep port | cut -d \" \" -f 2", "delta": "0:00:00.135695", "end": "2026-02-02 11:42:30.641940", "failed_when_result": true, "msg": "", "rc": 0, "start": "2026-02-02 11:42:30.506245", "stderr": "Error from server (NotFound): pods \"aodh-0\" not found", "stderr_lines": ["Error from server (NotFound): pods \"aodh-0\" not found"], "stdout": "", "stdout_lines": []} 2026-02-02 11:42:30,667 p=36171 u=zuul n=ansible | ...ignoring 2026-02-02 11:42:30,670 p=36171 u=zuul n=ansible | TASK [default-telemetry : Test podlevel TLS with NAD for MetricStorage] ******** 2026-02-02 11:42:30,670 p=36171 u=zuul n=ansible | Monday 02 February 2026 11:42:30 +0000 (0:00:00.319) 0:18:35.511 ******* 2026-02-02 11:42:30,687 p=36171 u=zuul n=ansible | included: /home/zuul/src/github.com/openstack-k8s-operators/telemetry-operator/ci/default-telemetry/tasks/test_podlevel_tls_with_nad_metricstorage.yml for localhost 2026-02-02 11:42:30,720 p=36171 u=zuul n=ansible | TASK [default-telemetry : Enable TLS at a pod level] *************************** 2026-02-02 11:42:30,720 p=36171 u=zuul n=ansible | Monday 02 February 2026 11:42:30 +0000 (0:00:00.049) 0:18:35.561 ******* 2026-02-02 11:42:31,217 p=36171 u=zuul n=ansible | changed: [localhost] 2026-02-02 11:42:31,221 p=36171 u=zuul n=ansible | TASK [default-telemetry : Set NAD for MetricStorage] *************************** 2026-02-02 11:42:31,221 p=36171 u=zuul n=ansible | Monday 02 February 2026 11:42:31 +0000 (0:00:00.501) 0:18:36.062 ******* 2026-02-02 11:42:31,716 p=36171 u=zuul n=ansible | changed: [localhost] 2026-02-02 11:42:31,720 p=36171 u=zuul n=ansible | TASK [default-telemetry : Wait until MetricStorage is ready] ******************* 2026-02-02 11:42:31,720 p=36171 u=zuul n=ansible | Monday 02 February 2026 11:42:31 +0000 (0:00:00.498) 0:18:36.561 ******* 2026-02-02 11:44:32,018 p=36171 u=zuul n=ansible | fatal: [localhost]: FAILED! => {"changed": true, "cmd": ["oc", "wait", "telemetry", "telemetry", "--for=condition=Ready", "--timeout=2m"], "delta": "0:02:00.107871", "end": "2026-02-02 11:44:31.995449", "msg": "non-zero return code", "rc": 1, "start": "2026-02-02 11:42:31.887578", "stderr": "error: timed out waiting for the condition on telemetries/telemetry", "stderr_lines": ["error: timed out waiting for the condition on telemetries/telemetry"], "stdout": "", "stdout_lines": []} 2026-02-02 11:44:32,018 p=36171 u=zuul n=ansible | ...ignoring 2026-02-02 11:44:32,022 p=36171 u=zuul n=ansible | TASK [default-telemetry : Get current time] ************************************ 2026-02-02 11:44:32,023 p=36171 u=zuul n=ansible | Monday 02 February 2026 11:44:32 +0000 (0:02:00.302) 0:20:36.863 ******* 2026-02-02 11:44:32,041 p=36171 u=zuul n=ansible | ok: [localhost] 2026-02-02 11:44:32,046 p=36171 u=zuul n=ansible | TASK [default-telemetry : Wait for new logs to be generated] ******************* 2026-02-02 11:44:32,046 p=36171 u=zuul n=ansible | Monday 02 February 2026 11:44:32 +0000 (0:00:00.023) 0:20:36.887 ******* 2026-02-02 11:46:32,249 p=36171 u=zuul n=ansible | ok: [localhost] 2026-02-02 11:46:32,253 p=36171 u=zuul n=ansible | TASK [default-telemetry : Get new telemetry-operator logs] ********************* 2026-02-02 11:46:32,253 p=36171 u=zuul n=ansible | Monday 02 February 2026 11:46:32 +0000 (0:02:00.207) 0:22:37.094 ******* 2026-02-02 11:46:32,614 p=36171 u=zuul n=ansible | changed: [localhost] 2026-02-02 11:46:32,617 p=36171 u=zuul n=ansible | TASK [default-telemetry : TEST Check that telemetry-operator logs don't include any errors when using TLS at a pod level OSPRH-14635] *** 2026-02-02 11:46:32,618 p=36171 u=zuul n=ansible | Monday 02 February 2026 11:46:32 +0000 (0:00:00.364) 0:22:37.459 ******* 2026-02-02 11:46:32,639 p=36171 u=zuul n=ansible | ok: [localhost] 2026-02-02 11:46:32,644 p=36171 u=zuul n=ansible | TASK [default-telemetry : Test Prometheus_secret values] *********************** 2026-02-02 11:46:32,644 p=36171 u=zuul n=ansible | Monday 02 February 2026 11:46:32 +0000 (0:00:00.026) 0:22:37.485 ******* 2026-02-02 11:46:32,660 p=36171 u=zuul n=ansible | included: /home/zuul/src/github.com/openstack-k8s-operators/telemetry-operator/ci/default-telemetry/tasks/test_prometheus_secret.yml for localhost 2026-02-02 11:46:32,692 p=36171 u=zuul n=ansible | TASK [default-telemetry : Get metric-storage-prometheus-endpoint secret values] *** 2026-02-02 11:46:32,692 p=36171 u=zuul n=ansible | Monday 02 February 2026 11:46:32 +0000 (0:00:00.048) 0:22:37.533 ******* 2026-02-02 11:46:33,050 p=36171 u=zuul n=ansible | ok: [localhost] 2026-02-02 11:46:33,055 p=36171 u=zuul n=ansible | TASK [default-telemetry : Show secret decoded values] ************************** 2026-02-02 11:46:33,055 p=36171 u=zuul n=ansible | Monday 02 February 2026 11:46:33 +0000 (0:00:00.363) 0:22:37.896 ******* 2026-02-02 11:46:33,074 p=36171 u=zuul n=ansible | ok: [localhost] => { "secret_output.stdout_lines": [ "ca_key=ca.crt", "ca_secret=ce**********vc", "host=metric-storage-prometheus.openstack.svc", "port=9090" ] } 2026-02-02 11:46:33,079 p=36171 u=zuul n=ansible | TASK [default-telemetry : Assert secret contains expected values] ************** 2026-02-02 11:46:33,079 p=36171 u=zuul n=ansible | Monday 02 February 2026 11:46:33 +0000 (0:00:00.023) 0:22:37.920 ******* 2026-02-02 11:46:33,100 p=36171 u=zuul n=ansible | ok: [localhost] => { "changed": false, "msg": "Secret metric-storage-prometheus-endpoint contains all required values" } 2026-02-02 11:46:33,115 p=36171 u=zuul n=ansible | PLAY RECAP ********************************************************************* 2026-02-02 11:46:33,116 p=36171 u=zuul n=ansible | localhost : ok=71 changed=30 unreachable=0 failed=0 skipped=1 rescued=0 ignored=9 2026-02-02 11:46:33,120 p=36171 u=zuul n=ansible | Monday 02 February 2026 11:46:33 +0000 (0:00:00.041) 0:22:37.961 ******* 2026-02-02 11:46:33,120 p=36171 u=zuul n=ansible | =============================================================================== 2026-02-02 11:46:33,121 p=36171 u=zuul n=ansible | default-telemetry : Wait for new logs to be generated ----------------- 120.43s 2026-02-02 11:46:33,121 p=36171 u=zuul n=ansible | default-telemetry : Wait until Autoscaling is ready ------------------- 120.35s 2026-02-02 11:46:33,121 p=36171 u=zuul n=ansible | default-telemetry : Wait until MetricStorage is ready ----------------- 120.34s 2026-02-02 11:46:33,121 p=36171 u=zuul n=ansible | default-telemetry : Wait for new logs to be generated ----------------- 120.31s 2026-02-02 11:46:33,121 p=36171 u=zuul n=ansible | default-telemetry : Wait until MetricStorage is ready ----------------- 120.30s 2026-02-02 11:46:33,121 p=36171 u=zuul n=ansible | default-telemetry : Wait until reconciliation finishes ---------------- 120.30s 2026-02-02 11:46:33,121 p=36171 u=zuul n=ansible | default-telemetry : Wait for new logs to be generated ----------------- 120.28s 2026-02-02 11:46:33,121 p=36171 u=zuul n=ansible | default-telemetry : Wait for new logs to be generated ----------------- 120.25s 2026-02-02 11:46:33,121 p=36171 u=zuul n=ansible | default-telemetry : Wait for new logs to be generated ----------------- 120.23s 2026-02-02 11:46:33,121 p=36171 u=zuul n=ansible | default-telemetry : Wait for new logs to be generated ----------------- 120.21s 2026-02-02 11:46:33,121 p=36171 u=zuul n=ansible | default-telemetry : Wait until Autoscaling and MetricStorage are ready -- 73.01s 2026-02-02 11:46:33,121 p=36171 u=zuul n=ansible | default-telemetry : Wait for the resources to be available ------------- 41.00s 2026-02-02 11:46:33,121 p=36171 u=zuul n=ansible | default-telemetry : Wait for the required resource to be created ------- 21.05s 2026-02-02 11:46:33,121 p=36171 u=zuul n=ansible | default-telemetry : Get Autoscaling condition values -------------------- 5.30s 2026-02-02 11:46:33,121 p=36171 u=zuul n=ansible | default-telemetry : Get MetricStorage condition values ------------------ 3.80s 2026-02-02 11:46:33,121 p=36171 u=zuul n=ansible | default-telemetry : Wait until telemetry is finished reconciling and its conditions are all True --- 0.55s 2026-02-02 11:46:33,121 p=36171 u=zuul n=ansible | default-telemetry : Enable Autoscaling ---------------------------------- 0.53s 2026-02-02 11:46:33,121 p=36171 u=zuul n=ansible | default-telemetry : Patch MetricStorage to use CustomMonitoringStack field --- 0.52s 2026-02-02 11:46:33,121 p=36171 u=zuul n=ansible | default-telemetry : Enable TLS at a pod level --------------------------- 0.50s 2026-02-02 11:46:33,121 p=36171 u=zuul n=ansible | default-telemetry : Set NAD for MetricStorage --------------------------- 0.50s 2026-02-02 11:46:33,121 p=36171 u=zuul n=ansible | default-telemetry : Patch Autoscaling to use a custom Prometheus instance --- 0.49s 2026-02-02 11:46:33,121 p=36171 u=zuul n=ansible | default-telemetry : Enable MetricStorage -------------------------------- 0.48s 2026-02-02 11:46:33,121 p=36171 u=zuul n=ansible | default-telemetry : Get telemetry-operator pod data --------------------- 0.43s 2026-02-02 11:46:33,121 p=36171 u=zuul n=ansible | default-telemetry : Get new telemetry-operator logs --------------------- 0.40s 2026-02-02 11:46:33,121 p=36171 u=zuul n=ansible | default-telemetry : Get Autoscaling condition types --------------------- 0.39s 2026-02-02 11:46:33,121 p=36171 u=zuul n=ansible | default-telemetry : Get new telemetry-operator logs --------------------- 0.39s 2026-02-02 11:46:33,121 p=36171 u=zuul n=ansible | default-telemetry : Get telemetry-operator failed container logs -------- 0.38s 2026-02-02 11:46:33,121 p=36171 u=zuul n=ansible | default-telemetry : Get new telemetry-operator logs --------------------- 0.38s 2026-02-02 11:46:33,121 p=36171 u=zuul n=ansible | default-telemetry : Get new telemetry-operator logs --------------------- 0.38s 2026-02-02 11:46:33,121 p=36171 u=zuul n=ansible | default-telemetry : Get telemetry-operator pod data --------------------- 0.36s 2026-02-02 11:46:33,121 p=36171 u=zuul n=ansible | default-telemetry : Get new telemetry-operator logs --------------------- 0.36s 2026-02-02 11:46:33,121 p=36171 u=zuul n=ansible | default-telemetry : Get metric-storage-prometheus-endpoint secret values --- 0.36s 2026-02-02 11:46:33,121 p=36171 u=zuul n=ansible | default-telemetry : Get telemetry-operator failed container logs -------- 0.36s 2026-02-02 11:46:33,121 p=36171 u=zuul n=ansible | default-telemetry : Create the COO subscription ------------------------- 0.35s 2026-02-02 11:46:33,122 p=36171 u=zuul n=ansible | default-telemetry : Get new telemetry-operator logs --------------------- 0.35s 2026-02-02 11:46:33,122 p=36171 u=zuul n=ansible | default-telemetry : Get MetricStorage condition types ------------------- 0.33s 2026-02-02 11:46:33,122 p=36171 u=zuul n=ansible | default-telemetry : TEST Check Prometheus host is set correctly in the aodh-evaluator container when using a custom Prometheus RHOSO-123466 --- 0.33s 2026-02-02 11:46:33,122 p=36171 u=zuul n=ansible | default-telemetry : TEST Check Prometheus port is set correctly in the aodh-evaluator container when using a custom Prometheus RHOSO-123467 --- 0.32s 2026-02-02 11:46:33,122 p=36171 u=zuul n=ansible | default-telemetry : Construct Autoscaling condition dictionary ---------- 0.17s 2026-02-02 11:46:33,122 p=36171 u=zuul n=ansible | default-telemetry : Construct MetricStorage condition dictionary -------- 0.16s 2026-02-02 11:46:33,122 p=36171 u=zuul n=ansible | default-telemetry : Check telemetry without COO and with Autoscaling and MetricStorage enabled --- 0.06s 2026-02-02 11:46:33,122 p=36171 u=zuul n=ansible | default-telemetry : Test CustomMonitoringStack -------------------------- 0.06s 2026-02-02 11:46:33,122 p=36171 u=zuul n=ansible | default-telemetry : Test podlevel TLS with NAD for MetricStorage -------- 0.05s 2026-02-02 11:46:33,122 p=36171 u=zuul n=ansible | default-telemetry : Test Prometheus_secret values ----------------------- 0.05s 2026-02-02 11:46:33,122 p=36171 u=zuul n=ansible | default-telemetry : Check telemetry with COO and with Autoscaling and MetricStorage enabled --- 0.04s 2026-02-02 11:46:33,122 p=36171 u=zuul n=ansible | default-telemetry : Assert secret contains expected values -------------- 0.04s 2026-02-02 11:46:33,122 p=36171 u=zuul n=ansible | default-telemetry : TEST Check that all MetricStorage conditions are as expected when COO isn't installed RHOSO-123461 --- 0.04s 2026-02-02 11:46:33,122 p=36171 u=zuul n=ansible | default-telemetry : Enable Autoscaling and MetricStorage ---------------- 0.04s 2026-02-02 11:46:33,122 p=36171 u=zuul n=ansible | default-telemetry : Get current time ------------------------------------ 0.04s 2026-02-02 11:46:33,122 p=36171 u=zuul n=ansible | default-telemetry : Check telemetry without COO and without Autoscaling and MetricStorage enabled --- 0.03s 2026-02-02 11:46:33,122 p=36171 u=zuul n=ansible | default-telemetry : TEST Check that all Autoscaling conditions are as expected when COO isn't installed RHOSO-123462 --- 0.03s 2026-02-02 11:46:33,122 p=36171 u=zuul n=ansible | default-telemetry : TEST Check that telemetry-operator logs don't include any errors after COO is installed RHOSO-123463 --- 0.03s 2026-02-02 11:46:33,122 p=36171 u=zuul n=ansible | default-telemetry : TEST Check telemetry-operator logs for errors after it's started RHOSO-123456 --- 0.03s 2026-02-02 11:46:33,122 p=36171 u=zuul n=ansible | default-telemetry : TEST Check that telemetry-operator logs include expected error logs when MetricStorage is enabled, but COO isn't installed RHOSO-123459 --- 0.03s 2026-02-02 11:46:33,122 p=36171 u=zuul n=ansible | default-telemetry : TEST Check that telemetry-operator logs don't include any errors when using TLS at a pod level OSPRH-14635 --- 0.03s 2026-02-02 11:46:33,122 p=36171 u=zuul n=ansible | default-telemetry : Enable Autoscaling and MetricStorage ---------------- 0.03s 2026-02-02 11:46:33,122 p=36171 u=zuul n=ansible | default-telemetry : Get telemetry-operator restart counts --------------- 0.03s 2026-02-02 11:46:33,122 p=36171 u=zuul n=ansible | default-telemetry : TEST Check that telemetry-operator logs don't include any errors when using a custom Prometheus instance for Autoscaling RHOSO-123465 --- 0.03s 2026-02-02 11:46:33,122 p=36171 u=zuul n=ansible | default-telemetry : Get telemetry-operator restart counts --------------- 0.02s 2026-02-02 11:46:33,122 p=36171 u=zuul n=ansible | default-telemetry : Output logs of failed container for debugging purposes --- 0.02s 2026-02-02 11:46:33,122 p=36171 u=zuul n=ansible | default-telemetry : Show secret decoded values -------------------------- 0.02s 2026-02-02 11:46:33,122 p=36171 u=zuul n=ansible | default-telemetry : Get current time ------------------------------------ 0.02s 2026-02-02 11:46:33,122 p=36171 u=zuul n=ansible | default-telemetry : Get current time ------------------------------------ 0.02s 2026-02-02 11:46:33,122 p=36171 u=zuul n=ansible | default-telemetry : Get current time ------------------------------------ 0.02s 2026-02-02 11:46:33,122 p=36171 u=zuul n=ansible | default-telemetry : Output logs of failed container for debugging purposes --- 0.02s 2026-02-02 11:46:33,122 p=36171 u=zuul n=ansible | default-telemetry : Get telemetry-operator container statuses ----------- 0.02s 2026-02-02 11:46:33,122 p=36171 u=zuul n=ansible | default-telemetry : Get current time ------------------------------------ 0.02s 2026-02-02 11:46:33,122 p=36171 u=zuul n=ansible | default-telemetry : Get current time ------------------------------------ 0.02s 2026-02-02 11:46:33,122 p=36171 u=zuul n=ansible | default-telemetry : TEST Check that telemetry-operator logs don't include errors when MetricStorage is enabled, but COO isn't installed RHOSO-123460 --- 0.02s 2026-02-02 11:46:33,122 p=36171 u=zuul n=ansible | default-telemetry : Get telemetry-operator container statuses ----------- 0.02s 2026-02-02 11:46:33,123 p=36171 u=zuul n=ansible | default-telemetry : TEST Check that telemetry-operator logs don't include any errors when using the CustomMonitoringStack RHOSO-123464 --- 0.02s