2026-02-02 13:42:57,526 p=36239 u=zuul n=ansible | [WARNING]: Found variable using reserved name: namespace

2026-02-02 13:42:57,526 p=36239 u=zuul n=ansible | PLAY [Execute the default-telemetry scenario] **********************************
2026-02-02 13:42:57,561 p=36239 u=zuul n=ansible | TASK [default-telemetry : Check telemetry without COO and without Autoscaling and MetricStorage enabled] ***
2026-02-02 13:42:57,561 p=36239 u=zuul n=ansible | Monday 02 February 2026  13:42:57 +0000 (0:00:00.049)       0:00:00.049 ******* 
2026-02-02 13:42:57,581 p=36239 u=zuul n=ansible | included: /home/zuul/src/github.com/openstack-k8s-operators/telemetry-operator/ci/default-telemetry/tasks/no_coo_no_autoscaling_no_metricstorage.yml for localhost
2026-02-02 13:42:57,588 p=36239 u=zuul n=ansible | TASK [default-telemetry : Wait until telemetry is finished reconciling and its conditions are all True] ***
2026-02-02 13:42:57,588 p=36239 u=zuul n=ansible | Monday 02 February 2026  13:42:57 +0000 (0:00:00.026)       0:00:00.076 ******* 
2026-02-02 13:42:58,105 p=36239 u=zuul n=ansible | changed: [localhost]
2026-02-02 13:42:58,109 p=36239 u=zuul n=ansible | TASK [default-telemetry : Get current time] ************************************
2026-02-02 13:42:58,109 p=36239 u=zuul n=ansible | Monday 02 February 2026  13:42:58 +0000 (0:00:00.521)       0:00:00.598 ******* 
2026-02-02 13:42:58,124 p=36239 u=zuul n=ansible | ok: [localhost]
2026-02-02 13:42:58,127 p=36239 u=zuul n=ansible | TASK [default-telemetry : Wait for new logs to be generated] *******************
2026-02-02 13:42:58,127 p=36239 u=zuul n=ansible | Monday 02 February 2026  13:42:58 +0000 (0:00:00.018)       0:00:00.616 ******* 
2026-02-02 13:44:58,449 p=36239 u=zuul n=ansible | ok: [localhost]
2026-02-02 13:44:58,453 p=36239 u=zuul n=ansible | TASK [default-telemetry : Get new telemetry-operator logs] *********************
2026-02-02 13:44:58,453 p=36239 u=zuul n=ansible | Monday 02 February 2026  13:44:58 +0000 (0:02:00.326)       0:02:00.942 ******* 
2026-02-02 13:44:58,866 p=36239 u=zuul n=ansible | changed: [localhost]
2026-02-02 13:44:58,871 p=36239 u=zuul n=ansible | TASK [default-telemetry : TEST Check telemetry-operator logs for errors after it's started
RHOSO-123456] ***
2026-02-02 13:44:58,871 p=36239 u=zuul n=ansible | Monday 02 February 2026  13:44:58 +0000 (0:00:00.417)       0:02:01.360 ******* 
2026-02-02 13:44:58,892 p=36239 u=zuul n=ansible | ok: [localhost]
2026-02-02 13:44:58,896 p=36239 u=zuul n=ansible | TASK [default-telemetry : Output found errors for debugging purposes] **********
2026-02-02 13:44:58,896 p=36239 u=zuul n=ansible | Monday 02 February 2026  13:44:58 +0000 (0:00:00.025)       0:02:01.385 ******* 
2026-02-02 13:44:58,909 p=36239 u=zuul n=ansible | skipping: [localhost]
2026-02-02 13:44:58,914 p=36239 u=zuul n=ansible | TASK [default-telemetry : Get telemetry-operator pod data] *********************
2026-02-02 13:44:58,914 p=36239 u=zuul n=ansible | Monday 02 February 2026  13:44:58 +0000 (0:00:00.017)       0:02:01.403 ******* 
2026-02-02 13:44:59,267 p=36239 u=zuul n=ansible | changed: [localhost]
2026-02-02 13:44:59,271 p=36239 u=zuul n=ansible | TASK [default-telemetry : Get telemetry-operator container statuses] ***********
2026-02-02 13:44:59,271 p=36239 u=zuul n=ansible | Monday 02 February 2026  13:44:59 +0000 (0:00:00.356)       0:02:01.760 ******* 
2026-02-02 13:44:59,284 p=36239 u=zuul n=ansible | ok: [localhost]
2026-02-02 13:44:59,288 p=36239 u=zuul n=ansible | TASK [default-telemetry : Get telemetry-operator restart counts] ***************
2026-02-02 13:44:59,288 p=36239 u=zuul n=ansible | Monday 02 February 2026  13:44:59 +0000 (0:00:00.017)       0:02:01.777 ******* 
2026-02-02 13:44:59,305 p=36239 u=zuul n=ansible | ok: [localhost]
2026-02-02 13:44:59,308 p=36239 u=zuul n=ansible | TASK [default-telemetry : Get telemetry-operator failed container logs] ********
2026-02-02 13:44:59,308 p=36239 u=zuul n=ansible | Monday 02 February 2026  13:44:59 +0000 (0:00:00.020)       0:02:01.797 ******* 
2026-02-02 13:44:59,641 p=36239 u=zuul n=ansible | fatal: [localhost]: FAILED! => {"changed": true, "cmd": ["oc", "logs", "-n", "openstack-operators", "-p", "-l", "openstack.org/operator-name=telemetry", "--tail=-1"], "delta": "0:00:00.134746", "end": "2026-02-02 13:44:59.618421", "msg": "non-zero return code", "rc": 1, "start": "2026-02-02 13:44:59.483675", "stderr": "Error from server (BadRequest): previous terminated container \"manager\" in pod \"telemetry-operator-controller-manager-5ff9b8ccfd-zh2nx\" not found", "stderr_lines": ["Error from server (BadRequest): previous terminated container \"manager\" in pod \"telemetry-operator-controller-manager-5ff9b8ccfd-zh2nx\" not found"], "stdout": "", "stdout_lines": []}
2026-02-02 13:44:59,641 p=36239 u=zuul n=ansible | ...ignoring
2026-02-02 13:44:59,645 p=36239 u=zuul n=ansible | TASK [default-telemetry : Output logs of failed container for debugging purposes] ***
2026-02-02 13:44:59,645 p=36239 u=zuul n=ansible | Monday 02 February 2026  13:44:59 +0000 (0:00:00.336)       0:02:02.134 ******* 
2026-02-02 13:44:59,664 p=36239 u=zuul n=ansible | ok: [localhost] => {
    "operator_logs_previous.stdout_lines": []
}
2026-02-02 13:44:59,668 p=36239 u=zuul n=ansible | TASK [default-telemetry : Enable Autoscaling and MetricStorage] ****************
2026-02-02 13:44:59,668 p=36239 u=zuul n=ansible | Monday 02 February 2026  13:44:59 +0000 (0:00:00.022)       0:02:02.156 ******* 
2026-02-02 13:44:59,680 p=36239 u=zuul n=ansible | included: /home/zuul/src/github.com/openstack-k8s-operators/telemetry-operator/ci/default-telemetry/tasks/enable_autoscaling_and_metricstorage.yml for localhost
2026-02-02 13:44:59,691 p=36239 u=zuul n=ansible | TASK [default-telemetry : Enable MetricStorage] ********************************
2026-02-02 13:44:59,691 p=36239 u=zuul n=ansible | Monday 02 February 2026  13:44:59 +0000 (0:00:00.023)       0:02:02.180 ******* 
2026-02-02 13:45:00,165 p=36239 u=zuul n=ansible | changed: [localhost]
2026-02-02 13:45:00,169 p=36239 u=zuul n=ansible | TASK [default-telemetry : Enable Autoscaling] **********************************
2026-02-02 13:45:00,169 p=36239 u=zuul n=ansible | Monday 02 February 2026  13:45:00 +0000 (0:00:00.477)       0:02:02.658 ******* 
2026-02-02 13:45:00,662 p=36239 u=zuul n=ansible | changed: [localhost]
2026-02-02 13:45:00,666 p=36239 u=zuul n=ansible | TASK [default-telemetry : Wait until reconciliation finishes] ******************
2026-02-02 13:45:00,666 p=36239 u=zuul n=ansible | Monday 02 February 2026  13:45:00 +0000 (0:00:00.497)       0:02:03.155 ******* 
2026-02-02 13:47:00,965 p=36239 u=zuul n=ansible | ok: [localhost]
2026-02-02 13:47:00,969 p=36239 u=zuul n=ansible | TASK [default-telemetry : Check telemetry without COO and with Autoscaling and MetricStorage enabled] ***
2026-02-02 13:47:00,970 p=36239 u=zuul n=ansible | Monday 02 February 2026  13:47:00 +0000 (0:02:00.303)       0:04:03.458 ******* 
2026-02-02 13:47:00,995 p=36239 u=zuul n=ansible | included: /home/zuul/src/github.com/openstack-k8s-operators/telemetry-operator/ci/default-telemetry/tasks/no_coo_autoscaling_metricstorage.yml for localhost
2026-02-02 13:47:01,009 p=36239 u=zuul n=ansible | TASK [default-telemetry : Get telemetry-operator pod data] *********************
2026-02-02 13:47:01,009 p=36239 u=zuul n=ansible | Monday 02 February 2026  13:47:01 +0000 (0:00:00.039)       0:04:03.498 ******* 
2026-02-02 13:47:01,342 p=36239 u=zuul n=ansible | changed: [localhost]
2026-02-02 13:47:01,345 p=36239 u=zuul n=ansible | TASK [default-telemetry : Get telemetry-operator container statuses] ***********
2026-02-02 13:47:01,346 p=36239 u=zuul n=ansible | Monday 02 February 2026  13:47:01 +0000 (0:00:00.336)       0:04:03.834 ******* 
2026-02-02 13:47:01,362 p=36239 u=zuul n=ansible | ok: [localhost]
2026-02-02 13:47:01,365 p=36239 u=zuul n=ansible | TASK [default-telemetry : Get telemetry-operator restart counts] ***************
2026-02-02 13:47:01,366 p=36239 u=zuul n=ansible | Monday 02 February 2026  13:47:01 +0000 (0:00:00.020)       0:04:03.854 ******* 
2026-02-02 13:47:01,384 p=36239 u=zuul n=ansible | ok: [localhost]
2026-02-02 13:47:01,388 p=36239 u=zuul n=ansible | TASK [default-telemetry : Get telemetry-operator failed container logs] ********
2026-02-02 13:47:01,388 p=36239 u=zuul n=ansible | Monday 02 February 2026  13:47:01 +0000 (0:00:00.022)       0:04:03.877 ******* 
2026-02-02 13:47:01,728 p=36239 u=zuul n=ansible | fatal: [localhost]: FAILED! => {"changed": true, "cmd": ["oc", "logs", "-n", "openstack-operators", "-p", "-l", "openstack.org/operator-name=telemetry", "--tail=-1"], "delta": "0:00:00.140894", "end": "2026-02-02 13:47:01.702181", "msg": "non-zero return code", "rc": 1, "start": "2026-02-02 13:47:01.561287", "stderr": "Error from server (BadRequest): previous terminated container \"manager\" in pod \"telemetry-operator-controller-manager-5ff9b8ccfd-zh2nx\" not found", "stderr_lines": ["Error from server (BadRequest): previous terminated container \"manager\" in pod \"telemetry-operator-controller-manager-5ff9b8ccfd-zh2nx\" not found"], "stdout": "", "stdout_lines": []}
2026-02-02 13:47:01,728 p=36239 u=zuul n=ansible | ...ignoring
2026-02-02 13:47:01,732 p=36239 u=zuul n=ansible | TASK [default-telemetry : Output logs of failed container for debugging purposes] ***
2026-02-02 13:47:01,732 p=36239 u=zuul n=ansible | Monday 02 February 2026  13:47:01 +0000 (0:00:00.344)       0:04:04.221 ******* 
2026-02-02 13:47:01,753 p=36239 u=zuul n=ansible | ok: [localhost] => {
    "operator_logs_previous.stdout_lines": []
}
2026-02-02 13:47:01,757 p=36239 u=zuul n=ansible | TASK [default-telemetry : Get current time] ************************************
2026-02-02 13:47:01,757 p=36239 u=zuul n=ansible | Monday 02 February 2026  13:47:01 +0000 (0:00:00.024)       0:04:04.246 ******* 
2026-02-02 13:47:01,773 p=36239 u=zuul n=ansible | ok: [localhost]
2026-02-02 13:47:01,777 p=36239 u=zuul n=ansible | TASK [default-telemetry : Wait for new logs to be generated] *******************
2026-02-02 13:47:01,778 p=36239 u=zuul n=ansible | Monday 02 February 2026  13:47:01 +0000 (0:00:00.020)       0:04:04.266 ******* 
2026-02-02 13:49:02,040 p=36239 u=zuul n=ansible | ok: [localhost]
2026-02-02 13:49:02,044 p=36239 u=zuul n=ansible | TASK [default-telemetry : Get new telemetry-operator logs] *********************
2026-02-02 13:49:02,044 p=36239 u=zuul n=ansible | Monday 02 February 2026  13:49:02 +0000 (0:02:00.266)       0:06:04.532 ******* 
2026-02-02 13:49:02,405 p=36239 u=zuul n=ansible | changed: [localhost]
2026-02-02 13:49:02,410 p=36239 u=zuul n=ansible | TASK [default-telemetry : TEST Check that telemetry-operator logs include expected error logs when MetricStorage is enabled, but COO isn't installed
RHOSO-123459] ***
2026-02-02 13:49:02,410 p=36239 u=zuul n=ansible | Monday 02 February 2026  13:49:02 +0000 (0:00:00.366)       0:06:04.899 ******* 
2026-02-02 13:49:02,430 p=36239 u=zuul n=ansible | ok: [localhost] => (item=INFO	Controllers.MetricStorage	Can't own MonitoringStack resource) => {
    "ansible_loop_var": "item",
    "changed": false,
    "item": "INFO\tControllers.MetricStorage\tCan't own MonitoringStack resource",
    "msg": "All assertions passed"
}
2026-02-02 13:49:02,436 p=36239 u=zuul n=ansible | TASK [default-telemetry : TEST Check that telemetry-operator logs don't include errors when MetricStorage is enabled, but COO isn't installed
RHOSO-123460] ***
2026-02-02 13:49:02,436 p=36239 u=zuul n=ansible | Monday 02 February 2026  13:49:02 +0000 (0:00:00.025)       0:06:04.925 ******* 
2026-02-02 13:49:02,454 p=36239 u=zuul n=ansible | fatal: [localhost]: FAILED! => {"ansible_facts": {"error_list": ["ERROR\tReconciler error\t{\"controller\": \"autoscaling\", \"controllerGroup\": \"telemetry.openstack.org\", \"controllerKind\": \"Autoscaling\", \"Autoscaling\": {\"name\":\"autoscaling\",\"namespace\":\"openstack\"}, \"namespace\": \"openstack\", \"name\": \"autoscaling\", \"reconcileID\": \"db7f1469-30a9-4488-828a-df05b6b47d16\", \"error\": \"ApplicationCredential secret not found: ac-aodh-secret\"}"]}, "changed": false, "failed_when_result": true}
2026-02-02 13:49:02,454 p=36239 u=zuul n=ansible | ...ignoring
2026-02-02 13:49:02,459 p=36239 u=zuul n=ansible | TASK [default-telemetry : Get MetricStorage condition types] *******************
2026-02-02 13:49:02,459 p=36239 u=zuul n=ansible | Monday 02 February 2026  13:49:02 +0000 (0:00:00.022)       0:06:04.947 ******* 
2026-02-02 13:49:02,802 p=36239 u=zuul n=ansible | changed: [localhost]
2026-02-02 13:49:02,806 p=36239 u=zuul n=ansible | TASK [default-telemetry : Get MetricStorage condition values] ******************
2026-02-02 13:49:02,806 p=36239 u=zuul n=ansible | Monday 02 February 2026  13:49:02 +0000 (0:00:00.347)       0:06:05.295 ******* 
2026-02-02 13:49:03,161 p=36239 u=zuul n=ansible | changed: [localhost] => (item=Ready)
2026-02-02 13:49:03,481 p=36239 u=zuul n=ansible | changed: [localhost] => (item=DashboardDatasourceReady)
2026-02-02 13:49:03,818 p=36239 u=zuul n=ansible | changed: [localhost] => (item=DashboardDefinitionReady)
2026-02-02 13:49:04,153 p=36239 u=zuul n=ansible | changed: [localhost] => (item=DashboardPluginReady)
2026-02-02 13:49:04,496 p=36239 u=zuul n=ansible | changed: [localhost] => (item=DashboardPrometheusRuleReady)
2026-02-02 13:49:04,876 p=36239 u=zuul n=ansible | changed: [localhost] => (item=MonitoringStackReady)
2026-02-02 13:49:05,216 p=36239 u=zuul n=ansible | changed: [localhost] => (item=NetworkAttachmentsReady)
2026-02-02 13:49:05,527 p=36239 u=zuul n=ansible | changed: [localhost] => (item=PrometheusReady)
2026-02-02 13:49:05,862 p=36239 u=zuul n=ansible | changed: [localhost] => (item=ScrapeConfigReady)
2026-02-02 13:49:06,182 p=36239 u=zuul n=ansible | changed: [localhost] => (item=TLSInputReady)
2026-02-02 13:49:06,190 p=36239 u=zuul n=ansible | TASK [default-telemetry : Construct MetricStorage condition dictionary] ********
2026-02-02 13:49:06,190 p=36239 u=zuul n=ansible | Monday 02 February 2026  13:49:06 +0000 (0:00:03.383)       0:06:08.678 ******* 
2026-02-02 13:49:06,221 p=36239 u=zuul n=ansible | ok: [localhost] => (item={'changed': True, 'stdout': 'Error occured when trying to own: customresourcedefinitions.apiextensions.k8s.io "monitoringstacks.monitoring.rhobs" not found', 'stderr': '', 'rc': 0, 'cmd': ['oc', 'get', 'metricstorage', 'metric-storage', '-o', 'jsonpath={.status.conditions[?(@.type == "Ready")].message}'], 'start': '2026-02-02 13:49:02.983076', 'end': '2026-02-02 13:49:03.128701', 'delta': '0:00:00.145625', 'msg': '', 'invocation': {'module_args': {'_raw_params': 'oc get metricstorage metric-storage -o jsonpath=\'{.status.conditions[?(@.type == "Ready")].message}\'', '_uses_shell': False, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': ['Error occured when trying to own: customresourcedefinitions.apiextensions.k8s.io "monitoringstacks.monitoring.rhobs" not found'], 'stderr_lines': [], 'failed': False, 'item': 'Ready', 'ansible_loop_var': 'item'})
2026-02-02 13:49:06,228 p=36239 u=zuul n=ansible | ok: [localhost] => (item={'changed': True, 'stdout': 'Dashboard Datasource not started', 'stderr': '', 'rc': 0, 'cmd': ['oc', 'get', 'metricstorage', 'metric-storage', '-o', 'jsonpath={.status.conditions[?(@.type == "DashboardDatasourceReady")].message}'], 'start': '2026-02-02 13:49:03.325807', 'end': '2026-02-02 13:49:03.454449', 'delta': '0:00:00.128642', 'msg': '', 'invocation': {'module_args': {'_raw_params': 'oc get metricstorage metric-storage -o jsonpath=\'{.status.conditions[?(@.type == "DashboardDatasourceReady")].message}\'', '_uses_shell': False, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': ['Dashboard Datasource not started'], 'stderr_lines': [], 'failed': False, 'item': 'DashboardDatasourceReady', 'ansible_loop_var': 'item'})
2026-02-02 13:49:06,236 p=36239 u=zuul n=ansible | ok: [localhost] => (item={'changed': True, 'stdout': 'Dashboard Definition not started', 'stderr': '', 'rc': 0, 'cmd': ['oc', 'get', 'metricstorage', 'metric-storage', '-o', 'jsonpath={.status.conditions[?(@.type == "DashboardDefinitionReady")].message}'], 'start': '2026-02-02 13:49:03.647110', 'end': '2026-02-02 13:49:03.789805', 'delta': '0:00:00.142695', 'msg': '', 'invocation': {'module_args': {'_raw_params': 'oc get metricstorage metric-storage -o jsonpath=\'{.status.conditions[?(@.type == "DashboardDefinitionReady")].message}\'', '_uses_shell': False, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': ['Dashboard Definition not started'], 'stderr_lines': [], 'failed': False, 'item': 'DashboardDefinitionReady', 'ansible_loop_var': 'item'})
2026-02-02 13:49:06,242 p=36239 u=zuul n=ansible | ok: [localhost] => (item={'changed': True, 'stdout': 'Dashboard Plugin not started', 'stderr': '', 'rc': 0, 'cmd': ['oc', 'get', 'metricstorage', 'metric-storage', '-o', 'jsonpath={.status.conditions[?(@.type == "DashboardPluginReady")].message}'], 'start': '2026-02-02 13:49:03.986052', 'end': '2026-02-02 13:49:04.128855', 'delta': '0:00:00.142803', 'msg': '', 'invocation': {'module_args': {'_raw_params': 'oc get metricstorage metric-storage -o jsonpath=\'{.status.conditions[?(@.type == "DashboardPluginReady")].message}\'', '_uses_shell': False, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': ['Dashboard Plugin not started'], 'stderr_lines': [], 'failed': False, 'item': 'DashboardPluginReady', 'ansible_loop_var': 'item'})
2026-02-02 13:49:06,249 p=36239 u=zuul n=ansible | ok: [localhost] => (item={'changed': True, 'stdout': 'Dashboard PrometheusRule not started', 'stderr': '', 'rc': 0, 'cmd': ['oc', 'get', 'metricstorage', 'metric-storage', '-o', 'jsonpath={.status.conditions[?(@.type == "DashboardPrometheusRuleReady")].message}'], 'start': '2026-02-02 13:49:04.326095', 'end': '2026-02-02 13:49:04.469137', 'delta': '0:00:00.143042', 'msg': '', 'invocation': {'module_args': {'_raw_params': 'oc get metricstorage metric-storage -o jsonpath=\'{.status.conditions[?(@.type == "DashboardPrometheusRuleReady")].message}\'', '_uses_shell': False, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': ['Dashboard PrometheusRule not started'], 'stderr_lines': [], 'failed': False, 'item': 'DashboardPrometheusRuleReady', 'ansible_loop_var': 'item'})
2026-02-02 13:49:06,257 p=36239 u=zuul n=ansible | ok: [localhost] => (item={'changed': True, 'stdout': 'Error occured when trying to own: customresourcedefinitions.apiextensions.k8s.io "monitoringstacks.monitoring.rhobs" not found', 'stderr': '', 'rc': 0, 'cmd': ['oc', 'get', 'metricstorage', 'metric-storage', '-o', 'jsonpath={.status.conditions[?(@.type == "MonitoringStackReady")].message}'], 'start': '2026-02-02 13:49:04.678403', 'end': '2026-02-02 13:49:04.847415', 'delta': '0:00:00.169012', 'msg': '', 'invocation': {'module_args': {'_raw_params': 'oc get metricstorage metric-storage -o jsonpath=\'{.status.conditions[?(@.type == "MonitoringStackReady")].message}\'', '_uses_shell': False, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': ['Error occured when trying to own: customresourcedefinitions.apiextensions.k8s.io "monitoringstacks.monitoring.rhobs" not found'], 'stderr_lines': [], 'failed': False, 'item': 'MonitoringStackReady', 'ansible_loop_var': 'item'})
2026-02-02 13:49:06,265 p=36239 u=zuul n=ansible | ok: [localhost] => (item={'changed': True, 'stdout': 'NetworkAttachments not started', 'stderr': '', 'rc': 0, 'cmd': ['oc', 'get', 'metricstorage', 'metric-storage', '-o', 'jsonpath={.status.conditions[?(@.type == "NetworkAttachmentsReady")].message}'], 'start': '2026-02-02 13:49:05.045481', 'end': '2026-02-02 13:49:05.188979', 'delta': '0:00:00.143498', 'msg': '', 'invocation': {'module_args': {'_raw_params': 'oc get metricstorage metric-storage -o jsonpath=\'{.status.conditions[?(@.type == "NetworkAttachmentsReady")].message}\'', '_uses_shell': False, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': ['NetworkAttachments not started'], 'stderr_lines': [], 'failed': False, 'item': 'NetworkAttachmentsReady', 'ansible_loop_var': 'item'})
2026-02-02 13:49:06,273 p=36239 u=zuul n=ansible | ok: [localhost] => (item={'changed': True, 'stdout': 'Prometheus not ready', 'stderr': '', 'rc': 0, 'cmd': ['oc', 'get', 'metricstorage', 'metric-storage', '-o', 'jsonpath={.status.conditions[?(@.type == "PrometheusReady")].message}'], 'start': '2026-02-02 13:49:05.364751', 'end': '2026-02-02 13:49:05.501086', 'delta': '0:00:00.136335', 'msg': '', 'invocation': {'module_args': {'_raw_params': 'oc get metricstorage metric-storage -o jsonpath=\'{.status.conditions[?(@.type == "PrometheusReady")].message}\'', '_uses_shell': False, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': ['Prometheus not ready'], 'stderr_lines': [], 'failed': False, 'item': 'PrometheusReady', 'ansible_loop_var': 'item'})
2026-02-02 13:49:06,279 p=36239 u=zuul n=ansible | ok: [localhost] => (item={'changed': True, 'stdout': 'ScrapeConfig not started', 'stderr': '', 'rc': 0, 'cmd': ['oc', 'get', 'metricstorage', 'metric-storage', '-o', 'jsonpath={.status.conditions[?(@.type == "ScrapeConfigReady")].message}'], 'start': '2026-02-02 13:49:05.685043', 'end': '2026-02-02 13:49:05.834772', 'delta': '0:00:00.149729', 'msg': '', 'invocation': {'module_args': {'_raw_params': 'oc get metricstorage metric-storage -o jsonpath=\'{.status.conditions[?(@.type == "ScrapeConfigReady")].message}\'', '_uses_shell': False, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': ['ScrapeConfig not started'], 'stderr_lines': [], 'failed': False, 'item': 'ScrapeConfigReady', 'ansible_loop_var': 'item'})
2026-02-02 13:49:06,286 p=36239 u=zuul n=ansible | ok: [localhost] => (item={'changed': True, 'stdout': 'Input data not checked', 'stderr': '', 'rc': 0, 'cmd': ['oc', 'get', 'metricstorage', 'metric-storage', '-o', 'jsonpath={.status.conditions[?(@.type == "TLSInputReady")].message}'], 'start': '2026-02-02 13:49:06.010586', 'end': '2026-02-02 13:49:06.157762', 'delta': '0:00:00.147176', 'msg': '', 'invocation': {'module_args': {'_raw_params': 'oc get metricstorage metric-storage -o jsonpath=\'{.status.conditions[?(@.type == "TLSInputReady")].message}\'', '_uses_shell': False, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': ['Input data not checked'], 'stderr_lines': [], 'failed': False, 'item': 'TLSInputReady', 'ansible_loop_var': 'item'})
2026-02-02 13:49:06,296 p=36239 u=zuul n=ansible | TASK [default-telemetry : TEST Check that all MetricStorage conditions are as expected when COO isn't installed
RHOSO-123461] ***
2026-02-02 13:49:06,296 p=36239 u=zuul n=ansible | Monday 02 February 2026  13:49:06 +0000 (0:00:00.106)       0:06:08.785 ******* 
2026-02-02 13:49:06,325 p=36239 u=zuul n=ansible | ok: [localhost] => (item={'key': 'MonitoringStackReady', 'value': 'Error occured when trying to own: customresourcedefinitions.apiextensions.k8s.io "monitoringstacks.monitoring.rhobs" not found'}) => {
    "ansible_loop_var": "item",
    "changed": false,
    "item": {
        "key": "MonitoringStackReady",
        "value": "Error occured when trying to own: customresourcedefinitions.apiextensions.k8s.io \"monitoringstacks.monitoring.rhobs\" not found"
    },
    "msg": "All assertions passed"
}
2026-02-02 13:49:06,329 p=36239 u=zuul n=ansible | ok: [localhost] => (item={'key': 'Ready', 'value': 'Error occured when trying to own: customresourcedefinitions.apiextensions.k8s.io "monitoringstacks.monitoring.rhobs" not found'}) => {
    "ansible_loop_var": "item",
    "changed": false,
    "item": {
        "key": "Ready",
        "value": "Error occured when trying to own: customresourcedefinitions.apiextensions.k8s.io \"monitoringstacks.monitoring.rhobs\" not found"
    },
    "msg": "All assertions passed"
}
2026-02-02 13:49:06,333 p=36239 u=zuul n=ansible | TASK [default-telemetry : Get Autoscaling condition types] *********************
2026-02-02 13:49:06,333 p=36239 u=zuul n=ansible | Monday 02 February 2026  13:49:06 +0000 (0:00:00.037)       0:06:08.822 ******* 
2026-02-02 13:49:06,669 p=36239 u=zuul n=ansible | changed: [localhost]
2026-02-02 13:49:06,673 p=36239 u=zuul n=ansible | TASK [default-telemetry : Get Autoscaling condition values] ********************
2026-02-02 13:49:06,673 p=36239 u=zuul n=ansible | Monday 02 February 2026  13:49:06 +0000 (0:00:00.340)       0:06:09.162 ******* 
2026-02-02 13:49:07,045 p=36239 u=zuul n=ansible | changed: [localhost] => (item=Ready)
2026-02-02 13:49:07,399 p=36239 u=zuul n=ansible | changed: [localhost] => (item=DBReady)
2026-02-02 13:49:07,727 p=36239 u=zuul n=ansible | changed: [localhost] => (item=DBSyncReady)
2026-02-02 13:49:08,112 p=36239 u=zuul n=ansible | changed: [localhost] => (item=DeploymentReady)
2026-02-02 13:49:08,455 p=36239 u=zuul n=ansible | changed: [localhost] => (item=HeatReady)
2026-02-02 13:49:08,780 p=36239 u=zuul n=ansible | changed: [localhost] => (item=KeystoneEndpointReady)
2026-02-02 13:49:09,111 p=36239 u=zuul n=ansible | changed: [localhost] => (item=KeystoneServiceReady)
2026-02-02 13:49:09,418 p=36239 u=zuul n=ansible | changed: [localhost] => (item=MariaDBAccountReady)
2026-02-02 13:49:09,743 p=36239 u=zuul n=ansible | changed: [localhost] => (item=MemcachedReady)
2026-02-02 13:49:10,074 p=36239 u=zuul n=ansible | changed: [localhost] => (item=RabbitMqTransportURLReady)
2026-02-02 13:49:10,420 p=36239 u=zuul n=ansible | changed: [localhost] => (item=RoleBindingReady)
2026-02-02 13:49:10,800 p=36239 u=zuul n=ansible | changed: [localhost] => (item=RoleReady)
2026-02-02 13:49:11,127 p=36239 u=zuul n=ansible | changed: [localhost] => (item=ServiceAccountReady)
2026-02-02 13:49:11,448 p=36239 u=zuul n=ansible | changed: [localhost] => (item=ServiceConfigReady)
2026-02-02 13:49:11,769 p=36239 u=zuul n=ansible | changed: [localhost] => (item=TLSInputReady)
2026-02-02 13:49:11,777 p=36239 u=zuul n=ansible | TASK [default-telemetry : Construct Autoscaling condition dictionary] **********
2026-02-02 13:49:11,777 p=36239 u=zuul n=ansible | Monday 02 February 2026  13:49:11 +0000 (0:00:05.103)       0:06:14.266 ******* 
2026-02-02 13:49:11,807 p=36239 u=zuul n=ansible | ok: [localhost] => (item={'changed': True, 'stdout': 'Service config create error occurred ApplicationCredential secret not found: ac-aodh-secret', 'stderr': '', 'rc': 0, 'cmd': ['oc', 'get', 'autoscaling', 'autoscaling', '-o', 'jsonpath={.status.conditions[?(@.type == "Ready")].message}'], 'start': '2026-02-02 13:49:06.844010', 'end': '2026-02-02 13:49:07.018803', 'delta': '0:00:00.174793', 'msg': '', 'invocation': {'module_args': {'_raw_params': 'oc get autoscaling autoscaling -o jsonpath=\'{.status.conditions[?(@.type == "Ready")].message}\'', '_uses_shell': False, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': ['Service config create error occurred ApplicationCredential secret not found: ac-aodh-secret'], 'stderr_lines': [], 'failed': False, 'item': 'Ready', 'ansible_loop_var': 'item'})
2026-02-02 13:49:11,815 p=36239 u=zuul n=ansible | ok: [localhost] => (item={'changed': True, 'stdout': 'DB create completed', 'stderr': '', 'rc': 0, 'cmd': ['oc', 'get', 'autoscaling', 'autoscaling', '-o', 'jsonpath={.status.conditions[?(@.type == "DBReady")].message}'], 'start': '2026-02-02 13:49:07.224026', 'end': '2026-02-02 13:49:07.373951', 'delta': '0:00:00.149925', 'msg': '', 'invocation': {'module_args': {'_raw_params': 'oc get autoscaling autoscaling -o jsonpath=\'{.status.conditions[?(@.type == "DBReady")].message}\'', '_uses_shell': False, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': ['DB create completed'], 'stderr_lines': [], 'failed': False, 'item': 'DBReady', 'ansible_loop_var': 'item'})
2026-02-02 13:49:11,823 p=36239 u=zuul n=ansible | ok: [localhost] => (item={'changed': True, 'stdout': 'DBsync not started', 'stderr': '', 'rc': 0, 'cmd': ['oc', 'get', 'autoscaling', 'autoscaling', '-o', 'jsonpath={.status.conditions[?(@.type == "DBSyncReady")].message}'], 'start': '2026-02-02 13:49:07.544924', 'end': '2026-02-02 13:49:07.677064', 'delta': '0:00:00.132140', 'msg': '', 'invocation': {'module_args': {'_raw_params': 'oc get autoscaling autoscaling -o jsonpath=\'{.status.conditions[?(@.type == "DBSyncReady")].message}\'', '_uses_shell': False, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': ['DBsync not started'], 'stderr_lines': [], 'failed': False, 'item': 'DBSyncReady', 'ansible_loop_var': 'item'})
2026-02-02 13:49:11,832 p=36239 u=zuul n=ansible | ok: [localhost] => (item={'changed': True, 'stdout': 'Deployment not started', 'stderr': '', 'rc': 0, 'cmd': ['oc', 'get', 'autoscaling', 'autoscaling', '-o', 'jsonpath={.status.conditions[?(@.type == "DeploymentReady")].message}'], 'start': '2026-02-02 13:49:07.901069', 'end': '2026-02-02 13:49:08.077426', 'delta': '0:00:00.176357', 'msg': '', 'invocation': {'module_args': {'_raw_params': 'oc get autoscaling autoscaling -o jsonpath=\'{.status.conditions[?(@.type == "DeploymentReady")].message}\'', '_uses_shell': False, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': ['Deployment not started'], 'stderr_lines': [], 'failed': False, 'item': 'DeploymentReady', 'ansible_loop_var': 'item'})
2026-02-02 13:49:11,840 p=36239 u=zuul n=ansible | ok: [localhost] => (item={'changed': True, 'stdout': 'Setup complete', 'stderr': '', 'rc': 0, 'cmd': ['oc', 'get', 'autoscaling', 'autoscaling', '-o', 'jsonpath={.status.conditions[?(@.type == "HeatReady")].message}'], 'start': '2026-02-02 13:49:08.292142', 'end': '2026-02-02 13:49:08.423857', 'delta': '0:00:00.131715', 'msg': '', 'invocation': {'module_args': {'_raw_params': 'oc get autoscaling autoscaling -o jsonpath=\'{.status.conditions[?(@.type == "HeatReady")].message}\'', '_uses_shell': False, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': ['Setup complete'], 'stderr_lines': [], 'failed': False, 'item': 'HeatReady', 'ansible_loop_var': 'item'})
2026-02-02 13:49:11,848 p=36239 u=zuul n=ansible | ok: [localhost] => (item={'changed': True, 'stdout': '', 'stderr': '', 'rc': 0, 'cmd': ['oc', 'get', 'autoscaling', 'autoscaling', '-o', 'jsonpath={.status.conditions[?(@.type == "KeystoneEndpointReady")].message}'], 'start': '2026-02-02 13:49:08.611566', 'end': '2026-02-02 13:49:08.749453', 'delta': '0:00:00.137887', 'msg': '', 'invocation': {'module_args': {'_raw_params': 'oc get autoscaling autoscaling -o jsonpath=\'{.status.conditions[?(@.type == "KeystoneEndpointReady")].message}\'', '_uses_shell': False, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': [], 'stderr_lines': [], 'failed': False, 'item': 'KeystoneEndpointReady', 'ansible_loop_var': 'item'})
2026-02-02 13:49:11,855 p=36239 u=zuul n=ansible | ok: [localhost] => (item={'changed': True, 'stdout': '', 'stderr': '', 'rc': 0, 'cmd': ['oc', 'get', 'autoscaling', 'autoscaling', '-o', 'jsonpath={.status.conditions[?(@.type == "KeystoneServiceReady")].message}'], 'start': '2026-02-02 13:49:08.930132', 'end': '2026-02-02 13:49:09.082028', 'delta': '0:00:00.151896', 'msg': '', 'invocation': {'module_args': {'_raw_params': 'oc get autoscaling autoscaling -o jsonpath=\'{.status.conditions[?(@.type == "KeystoneServiceReady")].message}\'', '_uses_shell': False, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': [], 'stderr_lines': [], 'failed': False, 'item': 'KeystoneServiceReady', 'ansible_loop_var': 'item'})
2026-02-02 13:49:11,863 p=36239 u=zuul n=ansible | ok: [localhost] => (item={'changed': True, 'stdout': 'MariaDBAccount creation complete', 'stderr': '', 'rc': 0, 'cmd': ['oc', 'get', 'autoscaling', 'autoscaling', '-o', 'jsonpath={.status.conditions[?(@.type == "MariaDBAccountReady")].message}'], 'start': '2026-02-02 13:49:09.265799', 'end': '2026-02-02 13:49:09.389614', 'delta': '0:00:00.123815', 'msg': '', 'invocation': {'module_args': {'_raw_params': 'oc get autoscaling autoscaling -o jsonpath=\'{.status.conditions[?(@.type == "MariaDBAccountReady")].message}\'', '_uses_shell': False, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': ['MariaDBAccount creation complete'], 'stderr_lines': [], 'failed': False, 'item': 'MariaDBAccountReady', 'ansible_loop_var': 'item'})
2026-02-02 13:49:11,870 p=36239 u=zuul n=ansible | ok: [localhost] => (item={'changed': True, 'stdout': ' Memcached instance has been provisioned', 'stderr': '', 'rc': 0, 'cmd': ['oc', 'get', 'autoscaling', 'autoscaling', '-o', 'jsonpath={.status.conditions[?(@.type == "MemcachedReady")].message}'], 'start': '2026-02-02 13:49:09.572796', 'end': '2026-02-02 13:49:09.714004', 'delta': '0:00:00.141208', 'msg': '', 'invocation': {'module_args': {'_raw_params': 'oc get autoscaling autoscaling -o jsonpath=\'{.status.conditions[?(@.type == "MemcachedReady")].message}\'', '_uses_shell': False, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': [' Memcached instance has been provisioned'], 'stderr_lines': [], 'failed': False, 'item': 'MemcachedReady', 'ansible_loop_var': 'item'})
2026-02-02 13:49:11,878 p=36239 u=zuul n=ansible | ok: [localhost] => (item={'changed': True, 'stdout': 'RabbitMqTransportURL successfully created', 'stderr': '', 'rc': 0, 'cmd': ['oc', 'get', 'autoscaling', 'autoscaling', '-o', 'jsonpath={.status.conditions[?(@.type == "RabbitMqTransportURLReady")].message}'], 'start': '2026-02-02 13:49:09.900945', 'end': '2026-02-02 13:49:10.044656', 'delta': '0:00:00.143711', 'msg': '', 'invocation': {'module_args': {'_raw_params': 'oc get autoscaling autoscaling -o jsonpath=\'{.status.conditions[?(@.type == "RabbitMqTransportURLReady")].message}\'', '_uses_shell': False, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': ['RabbitMqTransportURL successfully created'], 'stderr_lines': [], 'failed': False, 'item': 'RabbitMqTransportURLReady', 'ansible_loop_var': 'item'})
2026-02-02 13:49:11,885 p=36239 u=zuul n=ansible | ok: [localhost] => (item={'changed': True, 'stdout': 'RoleBinding created', 'stderr': '', 'rc': 0, 'cmd': ['oc', 'get', 'autoscaling', 'autoscaling', '-o', 'jsonpath={.status.conditions[?(@.type == "RoleBindingReady")].message}'], 'start': '2026-02-02 13:49:10.233159', 'end': '2026-02-02 13:49:10.388597', 'delta': '0:00:00.155438', 'msg': '', 'invocation': {'module_args': {'_raw_params': 'oc get autoscaling autoscaling -o jsonpath=\'{.status.conditions[?(@.type == "RoleBindingReady")].message}\'', '_uses_shell': False, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': ['RoleBinding created'], 'stderr_lines': [], 'failed': False, 'item': 'RoleBindingReady', 'ansible_loop_var': 'item'})
2026-02-02 13:49:11,893 p=36239 u=zuul n=ansible | ok: [localhost] => (item={'changed': True, 'stdout': 'Role created', 'stderr': '', 'rc': 0, 'cmd': ['oc', 'get', 'autoscaling', 'autoscaling', '-o', 'jsonpath={.status.conditions[?(@.type == "RoleReady")].message}'], 'start': '2026-02-02 13:49:10.620415', 'end': '2026-02-02 13:49:10.771359', 'delta': '0:00:00.150944', 'msg': '', 'invocation': {'module_args': {'_raw_params': 'oc get autoscaling autoscaling -o jsonpath=\'{.status.conditions[?(@.type == "RoleReady")].message}\'', '_uses_shell': False, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': ['Role created'], 'stderr_lines': [], 'failed': False, 'item': 'RoleReady', 'ansible_loop_var': 'item'})
2026-02-02 13:49:11,900 p=36239 u=zuul n=ansible | ok: [localhost] => (item={'changed': True, 'stdout': 'ServiceAccount created', 'stderr': '', 'rc': 0, 'cmd': ['oc', 'get', 'autoscaling', 'autoscaling', '-o', 'jsonpath={.status.conditions[?(@.type == "ServiceAccountReady")].message}'], 'start': '2026-02-02 13:49:10.968365', 'end': '2026-02-02 13:49:11.099889', 'delta': '0:00:00.131524', 'msg': '', 'invocation': {'module_args': {'_raw_params': 'oc get autoscaling autoscaling -o jsonpath=\'{.status.conditions[?(@.type == "ServiceAccountReady")].message}\'', '_uses_shell': False, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': ['ServiceAccount created'], 'stderr_lines': [], 'failed': False, 'item': 'ServiceAccountReady', 'ansible_loop_var': 'item'})
2026-02-02 13:49:11,908 p=36239 u=zuul n=ansible | ok: [localhost] => (item={'changed': True, 'stdout': 'Service config create error occurred ApplicationCredential secret not found: ac-aodh-secret', 'stderr': '', 'rc': 0, 'cmd': ['oc', 'get', 'autoscaling', 'autoscaling', '-o', 'jsonpath={.status.conditions[?(@.type == "ServiceConfigReady")].message}'], 'start': '2026-02-02 13:49:11.285468', 'end': '2026-02-02 13:49:11.422211', 'delta': '0:00:00.136743', 'msg': '', 'invocation': {'module_args': {'_raw_params': 'oc get autoscaling autoscaling -o jsonpath=\'{.status.conditions[?(@.type == "ServiceConfigReady")].message}\'', '_uses_shell': False, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': ['Service config create error occurred ApplicationCredential secret not found: ac-aodh-secret'], 'stderr_lines': [], 'failed': False, 'item': 'ServiceConfigReady', 'ansible_loop_var': 'item'})
2026-02-02 13:49:11,916 p=36239 u=zuul n=ansible | ok: [localhost] => (item={'changed': True, 'stdout': 'Input data not checked', 'stderr': '', 'rc': 0, 'cmd': ['oc', 'get', 'autoscaling', 'autoscaling', '-o', 'jsonpath={.status.conditions[?(@.type == "TLSInputReady")].message}'], 'start': '2026-02-02 13:49:11.604822', 'end': '2026-02-02 13:49:11.745398', 'delta': '0:00:00.140576', 'msg': '', 'invocation': {'module_args': {'_raw_params': 'oc get autoscaling autoscaling -o jsonpath=\'{.status.conditions[?(@.type == "TLSInputReady")].message}\'', '_uses_shell': False, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': ['Input data not checked'], 'stderr_lines': [], 'failed': False, 'item': 'TLSInputReady', 'ansible_loop_var': 'item'})
2026-02-02 13:49:11,929 p=36239 u=zuul n=ansible | TASK [default-telemetry : TEST Check that all Autoscaling conditions are as expected when COO isn't installed
RHOSO-123462] ***
2026-02-02 13:49:11,929 p=36239 u=zuul n=ansible | Monday 02 February 2026  13:49:11 +0000 (0:00:00.152)       0:06:14.418 ******* 
2026-02-02 13:49:11,949 p=36239 u=zuul n=ansible | failed: [localhost] (item={'key': 'Ready', 'value': 'Setup complete'}) => {
    "ansible_loop_var": "item",
    "assertion": "conditions[item.key] == item.value",
    "changed": false,
    "evaluated_to": false,
    "item": {
        "key": "Ready",
        "value": "Setup complete"
    },
    "msg": "Expected Ready condition field to be Setup complete, not {'results': [{'changed': True, 'stdout': 'Service config create error occurred ApplicationCredential secret not found: ac-aodh-secret', 'stderr': '', 'rc': 0, 'cmd': ['oc', 'get', 'autoscaling', 'autoscaling', '-o', 'jsonpath={.status.conditions[?(@.type == \"Ready\")].message}'], 'start': '2026-02-02 13:49:06.844010', 'end': '2026-02-02 13:49:07.018803', 'delta': '0:00:00.174793', 'msg': '', 'invocation': {'module_args': {'_raw_params': 'oc get autoscaling autoscaling -o jsonpath=\\'{.status.conditions[?(@.type == \"Ready\")].message}\\'', '_uses_shell': False, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': ['Service config create error occurred ApplicationCredential secret not found: ac-aodh-secret'], 'stderr_lines': [], 'failed': False, 'item': 'Ready', 'ansible_loop_var': 'item'}, {'changed': True, 'stdout': 'DB create completed', 'stderr': '', 'rc': 0, 'cmd': ['oc', 'get', 'autoscaling', 'autoscaling', '-o', 'jsonpath={.status.conditions[?(@.type == \"DBReady\")].message}'], 'start': '2026-02-02 13:49:07.224026', 'end': '2026-02-02 13:49:07.373951', 'delta': '0:00:00.149925', 'msg': '', 'invocation': {'module_args': {'_raw_params': 'oc get autoscaling autoscaling -o jsonpath=\\'{.status.conditions[?(@.type == \"DBReady\")].message}\\'', '_uses_shell': False, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': ['DB create completed'], 'stderr_lines': [], 'failed': False, 'item': 'DBReady', 'ansible_loop_var': 'item'}, {'changed': True, 'stdout': 'DBsync not started', 'stderr': '', 'rc': 0, 'cmd': ['oc', 'get', 'autoscaling', 'autoscaling', '-o', 'jsonpath={.status.conditions[?(@.type == \"DBSyncReady\")].message}'], 'start': '2026-02-02 13:49:07.544924', 'end': '2026-02-02 13:49:07.677064', 'delta': '0:00:00.132140', 'msg': '', 'invocation': {'module_args': {'_raw_params': 'oc get autoscaling autoscaling -o jsonpath=\\'{.status.conditions[?(@.type == \"DBSyncReady\")].message}\\'', '_uses_shell': False, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': ['DBsync not started'], 'stderr_lines': [], 'failed': False, 'item': 'DBSyncReady', 'ansible_loop_var': 'item'}, {'changed': True, 'stdout': 'Deployment not started', 'stderr': '', 'rc': 0, 'cmd': ['oc', 'get', 'autoscaling', 'autoscaling', '-o', 'jsonpath={.status.conditions[?(@.type == \"DeploymentReady\")].message}'], 'start': '2026-02-02 13:49:07.901069', 'end': '2026-02-02 13:49:08.077426', 'delta': '0:00:00.176357', 'msg': '', 'invocation': {'module_args': {'_raw_params': 'oc get autoscaling autoscaling -o jsonpath=\\'{.status.conditions[?(@.type == \"DeploymentReady\")].message}\\'', '_uses_shell': False, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': ['Deployment not started'], 'stderr_lines': [], 'failed': False, 'item': 'DeploymentReady', 'ansible_loop_var': 'item'}, {'changed': True, 'stdout': 'Setup complete', 'stderr': '', 'rc': 0, 'cmd': ['oc', 'get', 'autoscaling', 'autoscaling', '-o', 'jsonpath={.status.conditions[?(@.type == \"HeatReady\")].message}'], 'start': '2026-02-02 13:49:08.292142', 'end': '2026-02-02 13:49:08.423857', 'delta': '0:00:00.131715', 'msg': '', 'invocation': {'module_args': {'_raw_params': 'oc get autoscaling autoscaling -o jsonpath=\\'{.status.conditions[?(@.type == \"HeatReady\")].message}\\'', '_uses_shell': False, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': ['Setup complete'], 'stderr_lines': [], 'failed': False, 'item': 'HeatReady', 'ansible_loop_var': 'item'}, {'changed': True, 'stdout': '', 'stderr': '', 'rc': 0, 'cmd': ['oc', 'get', 'autoscaling', 'autoscaling', '-o', 'jsonpath={.status.conditions[?(@.type == \"KeystoneEndpointReady\")].message}'], 'start': '2026-02-02 13:49:08.611566', 'end': '2026-02-02 13:49:08.749453', 'delta': '0:00:00.137887', 'msg': '', 'invocation': {'module_args': {'_raw_params': 'oc get autoscaling autoscaling -o jsonpath=\\'{.status.conditions[?(@.type == \"KeystoneEndpointReady\")].message}\\'', '_uses_shell': False, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': [], 'stderr_lines': [], 'failed': False, 'item': 'KeystoneEndpointReady', 'ansible_loop_var': 'item'}, {'changed': True, 'stdout': '', 'stderr': '', 'rc': 0, 'cmd': ['oc', 'get', 'autoscaling', 'autoscaling', '-o', 'jsonpath={.status.conditions[?(@.type == \"KeystoneServiceReady\")].message}'], 'start': '2026-02-02 13:49:08.930132', 'end': '2026-02-02 13:49:09.082028', 'delta': '0:00:00.151896', 'msg': '', 'invocation': {'module_args': {'_raw_params': 'oc get autoscaling autoscaling -o jsonpath=\\'{.status.conditions[?(@.type == \"KeystoneServiceReady\")].message}\\'', '_uses_shell': False, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': [], 'stderr_lines': [], 'failed': False, 'item': 'KeystoneServiceReady', 'ansible_loop_var': 'item'}, {'changed': True, 'stdout': 'MariaDBAccount creation complete', 'stderr': '', 'rc': 0, 'cmd': ['oc', 'get', 'autoscaling', 'autoscaling', '-o', 'jsonpath={.status.conditions[?(@.type == \"MariaDBAccountReady\")].message}'], 'start': '2026-02-02 13:49:09.265799', 'end': '2026-02-02 13:49:09.389614', 'delta': '0:00:00.123815', 'msg': '', 'invocation': {'module_args': {'_raw_params': 'oc get autoscaling autoscaling -o jsonpath=\\'{.status.conditions[?(@.type == \"MariaDBAccountReady\")].message}\\'', '_uses_shell': False, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': ['MariaDBAccount creation complete'], 'stderr_lines': [], 'failed': False, 'item': 'MariaDBAccountReady', 'ansible_loop_var': 'item'}, {'changed': True, 'stdout': ' Memcached instance has been provisioned', 'stderr': '', 'rc': 0, 'cmd': ['oc', 'get', 'autoscaling', 'autoscaling', '-o', 'jsonpath={.status.conditions[?(@.type == \"MemcachedReady\")].message}'], 'start': '2026-02-02 13:49:09.572796', 'end': '2026-02-02 13:49:09.714004', 'delta': '0:00:00.141208', 'msg': '', 'invocation': {'module_args': {'_raw_params': 'oc get autoscaling autoscaling -o jsonpath=\\'{.status.conditions[?(@.type == \"MemcachedReady\")].message}\\'', '_uses_shell': False, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': [' Memcached instance has been provisioned'], 'stderr_lines': [], 'failed': False, 'item': 'MemcachedReady', 'ansible_loop_var': 'item'}, {'changed': True, 'stdout': 'RabbitMqTransportURL successfully created', 'stderr': '', 'rc': 0, 'cmd': ['oc', 'get', 'autoscaling', 'autoscaling', '-o', 'jsonpath={.status.conditions[?(@.type == \"RabbitMqTransportURLReady\")].message}'], 'start': '2026-02-02 13:49:09.900945', 'end': '2026-02-02 13:49:10.044656', 'delta': '0:00:00.143711', 'msg': '', 'invocation': {'module_args': {'_raw_params': 'oc get autoscaling autoscaling -o jsonpath=\\'{.status.conditions[?(@.type == \"RabbitMqTransportURLReady\")].message}\\'', '_uses_shell': False, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': ['RabbitMqTransportURL successfully created'], 'stderr_lines': [], 'failed': False, 'item': 'RabbitMqTransportURLReady', 'ansible_loop_var': 'item'}, {'changed': True, 'stdout': 'RoleBinding created', 'stderr': '', 'rc': 0, 'cmd': ['oc', 'get', 'autoscaling', 'autoscaling', '-o', 'jsonpath={.status.conditions[?(@.type == \"RoleBindingReady\")].message}'], 'start': '2026-02-02 13:49:10.233159', 'end': '2026-02-02 13:49:10.388597', 'delta': '0:00:00.155438', 'msg': '', 'invocation': {'module_args': {'_raw_params': 'oc get autoscaling autoscaling -o jsonpath=\\'{.status.conditions[?(@.type == \"RoleBindingReady\")].message}\\'', '_uses_shell': False, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': ['RoleBinding created'], 'stderr_lines': [], 'failed': False, 'item': 'RoleBindingReady', 'ansible_loop_var': 'item'}, {'changed': True, 'stdout': 'Role created', 'stderr': '', 'rc': 0, 'cmd': ['oc', 'get', 'autoscaling', 'autoscaling', '-o', 'jsonpath={.status.conditions[?(@.type == \"RoleReady\")].message}'], 'start': '2026-02-02 13:49:10.620415', 'end': '2026-02-02 13:49:10.771359', 'delta': '0:00:00.150944', 'msg': '', 'invocation': {'module_args': {'_raw_params': 'oc get autoscaling autoscaling -o jsonpath=\\'{.status.conditions[?(@.type == \"RoleReady\")].message}\\'', '_uses_shell': False, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': ['Role created'], 'stderr_lines': [], 'failed': False, 'item': 'RoleReady', 'ansible_loop_var': 'item'}, {'changed': True, 'stdout': 'ServiceAccount created', 'stderr': '', 'rc': 0, 'cmd': ['oc', 'get', 'autoscaling', 'autoscaling', '-o', 'jsonpath={.status.conditions[?(@.type == \"ServiceAccountReady\")].message}'], 'start': '2026-02-02 13:49:10.968365', 'end': '2026-02-02 13:49:11.099889', 'delta': '0:00:00.131524', 'msg': '', 'invocation': {'module_args': {'_raw_params': 'oc get autoscaling autoscaling -o jsonpath=\\'{.status.conditions[?(@.type == \"ServiceAccountReady\")].message}\\'', '_uses_shell': False, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': ['ServiceAccount created'], 'stderr_lines': [], 'failed': False, 'item': 'ServiceAccountReady', 'ansible_loop_var': 'item'}, {'changed': True, 'stdout': 'Service config create error occurred ApplicationCredential secret not found: ac-aodh-secret', 'stderr': '', 'rc': 0, 'cmd': ['oc', 'get', 'autoscaling', 'autoscaling', '-o', 'jsonpath={.status.conditions[?(@.type == \"ServiceConfigReady\")].message}'], 'start': '2026-02-02 13:49:11.285468', 'end': '2026-02-02 13:49:11.422211', 'delta': '0:00:00.136743', 'msg': '', 'invocation': {'module_args': {'_raw_params': 'oc get autoscaling autoscaling -o jsonpath=\\'{.status.conditions[?(@.type == \"ServiceConfigReady\")].message}\\'', '_uses_shell': False, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': ['Service config create error occurred ApplicationCredential secret not found: ac-aodh-secret'], 'stderr_lines': [], 'failed': False, 'item': 'ServiceConfigReady', 'ansible_loop_var': 'item'}, {'changed': True, 'stdout': 'Input data not checked', 'stderr': '', 'rc': 0, 'cmd': ['oc', 'get', 'autoscaling', 'autoscaling', '-o', 'jsonpath={.status.conditions[?(@.type == \"TLSInputReady\")].message}'], 'start': '2026-02-02 13:49:11.604822', 'end': '2026-02-02 13:49:11.745398', 'delta': '0:00:00.140576', 'msg': '', 'invocation': {'module_args': {'_raw_params': 'oc get autoscaling autoscaling -o jsonpath=\\'{.status.conditions[?(@.type == \"TLSInputReady\")].message}\\'', '_uses_shell': False, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': ['Input data not checked'], 'stderr_lines': [], 'failed': False, 'item': 'TLSInputReady', 'ansible_loop_var': 'item'}], 'skipped': False, 'changed': True, 'msg': 'All items completed'}"
}
2026-02-02 13:49:11,950 p=36239 u=zuul n=ansible | ...ignoring
2026-02-02 13:49:11,954 p=36239 u=zuul n=ansible | TASK [default-telemetry : Enable Autoscaling and MetricStorage] ****************
2026-02-02 13:49:11,954 p=36239 u=zuul n=ansible | Monday 02 February 2026  13:49:11 +0000 (0:00:00.024)       0:06:14.443 ******* 
2026-02-02 13:49:11,966 p=36239 u=zuul n=ansible | included: /home/zuul/src/github.com/openstack-k8s-operators/telemetry-operator/ci/default-telemetry/tasks/../../create-coo-subscription.yaml for localhost
2026-02-02 13:49:11,989 p=36239 u=zuul n=ansible | TASK [default-telemetry : Create the COO subscription] *************************
2026-02-02 13:49:11,989 p=36239 u=zuul n=ansible | Monday 02 February 2026  13:49:11 +0000 (0:00:00.035)       0:06:14.478 ******* 
2026-02-02 13:49:12,317 p=36239 u=zuul n=ansible | changed: [localhost]
2026-02-02 13:49:12,321 p=36239 u=zuul n=ansible | TASK [default-telemetry : Wait for the required resource to be created] ********
2026-02-02 13:49:12,321 p=36239 u=zuul n=ansible | Monday 02 February 2026  13:49:12 +0000 (0:00:00.331)       0:06:14.810 ******* 
2026-02-02 13:49:33,381 p=36239 u=zuul n=ansible | changed: [localhost]
2026-02-02 13:49:33,386 p=36239 u=zuul n=ansible | TASK [default-telemetry : Wait for the resources to be available] **************
2026-02-02 13:49:33,386 p=36239 u=zuul n=ansible | Monday 02 February 2026  13:49:33 +0000 (0:00:21.064)       0:06:35.875 ******* 
2026-02-02 13:49:53,533 p=36239 u=zuul n=ansible | changed: [localhost]
2026-02-02 13:49:53,536 p=36239 u=zuul n=ansible | TASK [default-telemetry : Check telemetry with COO and with Autoscaling and MetricStorage enabled] ***
2026-02-02 13:49:53,536 p=36239 u=zuul n=ansible | Monday 02 February 2026  13:49:53 +0000 (0:00:20.150)       0:06:56.025 ******* 
2026-02-02 13:49:53,551 p=36239 u=zuul n=ansible | included: /home/zuul/src/github.com/openstack-k8s-operators/telemetry-operator/ci/default-telemetry/tasks/coo_autoscaling_metricstorage.yml for localhost
2026-02-02 13:49:53,575 p=36239 u=zuul n=ansible | TASK [default-telemetry : Wait until Autoscaling and MetricStorage are ready] ***
2026-02-02 13:49:53,575 p=36239 u=zuul n=ansible | Monday 02 February 2026  13:49:53 +0000 (0:00:00.038)       0:06:56.064 ******* 
2026-02-02 13:50:35,403 p=36239 u=zuul n=ansible | changed: [localhost]
2026-02-02 13:50:35,407 p=36239 u=zuul n=ansible | TASK [default-telemetry : Get current time] ************************************
2026-02-02 13:50:35,408 p=36239 u=zuul n=ansible | Monday 02 February 2026  13:50:35 +0000 (0:00:41.832)       0:07:37.896 ******* 
2026-02-02 13:50:35,425 p=36239 u=zuul n=ansible | ok: [localhost]
2026-02-02 13:50:35,429 p=36239 u=zuul n=ansible | TASK [default-telemetry : Wait for new logs to be generated] *******************
2026-02-02 13:50:35,429 p=36239 u=zuul n=ansible | Monday 02 February 2026  13:50:35 +0000 (0:00:00.021)       0:07:37.918 ******* 
2026-02-02 13:52:35,732 p=36239 u=zuul n=ansible | ok: [localhost]
2026-02-02 13:52:35,736 p=36239 u=zuul n=ansible | TASK [default-telemetry : Get new telemetry-operator logs] *********************
2026-02-02 13:52:35,736 p=36239 u=zuul n=ansible | Monday 02 February 2026  13:52:35 +0000 (0:02:00.307)       0:09:38.225 ******* 
2026-02-02 13:52:36,107 p=36239 u=zuul n=ansible | changed: [localhost]
2026-02-02 13:52:36,112 p=36239 u=zuul n=ansible | TASK [default-telemetry : TEST Check that telemetry-operator logs don't include any errors after COO is installed
RHOSO-123463] ***
2026-02-02 13:52:36,112 p=36239 u=zuul n=ansible | Monday 02 February 2026  13:52:36 +0000 (0:00:00.375)       0:09:38.601 ******* 
2026-02-02 13:52:36,130 p=36239 u=zuul n=ansible | fatal: [localhost]: FAILED! => {
    "assertion": "operator_logs.stdout | ansible.builtin.regex_findall('ERROR.*') | length == 0",
    "changed": false,
    "evaluated_to": false,
    "msg": "There are errors in the telemetry-operator logs ['ERROR\\tReconciler error\\t{\"controller\": \"autoscaling\", \"controllerGroup\": \"telemetry.openstack.org\", \"controllerKind\": \"Autoscaling\", \"Autoscaling\": {\"name\":\"autoscaling\",\"namespace\":\"openstack\"}, \"namespace\": \"openstack\", \"name\": \"autoscaling\", \"reconcileID\": \"408a8d1c-55e8-4c93-a2e9-afa83e876e16\", \"error\": \"ApplicationCredential secret not found: ac-aodh-secret\"}']"
}
2026-02-02 13:52:36,130 p=36239 u=zuul n=ansible | ...ignoring
2026-02-02 13:52:36,134 p=36239 u=zuul n=ansible | TASK [default-telemetry : Test CustomMonitoringStack] **************************
2026-02-02 13:52:36,134 p=36239 u=zuul n=ansible | Monday 02 February 2026  13:52:36 +0000 (0:00:00.022)       0:09:38.623 ******* 
2026-02-02 13:52:36,154 p=36239 u=zuul n=ansible | included: /home/zuul/src/github.com/openstack-k8s-operators/telemetry-operator/ci/default-telemetry/tasks/test_custom_monitoring_stack.yml for localhost
2026-02-02 13:52:36,180 p=36239 u=zuul n=ansible | TASK [default-telemetry : Patch MetricStorage to use CustomMonitoringStack field] ***
2026-02-02 13:52:36,180 p=36239 u=zuul n=ansible | Monday 02 February 2026  13:52:36 +0000 (0:00:00.045)       0:09:38.668 ******* 
2026-02-02 13:52:36,761 p=36239 u=zuul n=ansible | changed: [localhost]
2026-02-02 13:52:36,765 p=36239 u=zuul n=ansible | TASK [default-telemetry : Wait until MetricStorage is ready] *******************
2026-02-02 13:52:36,766 p=36239 u=zuul n=ansible | Monday 02 February 2026  13:52:36 +0000 (0:00:00.585)       0:09:39.254 ******* 
2026-02-02 13:54:37,080 p=36239 u=zuul n=ansible | fatal: [localhost]: FAILED! => {"changed": true, "cmd": ["oc", "wait", "telemetry", "telemetry", "--for=condition=Ready", "--timeout=2m"], "delta": "0:02:00.118539", "end": "2026-02-02 13:54:37.057303", "msg": "non-zero return code", "rc": 1, "start": "2026-02-02 13:52:36.938764", "stderr": "error: timed out waiting for the condition on telemetries/telemetry", "stderr_lines": ["error: timed out waiting for the condition on telemetries/telemetry"], "stdout": "", "stdout_lines": []}
2026-02-02 13:54:37,080 p=36239 u=zuul n=ansible | ...ignoring
2026-02-02 13:54:37,084 p=36239 u=zuul n=ansible | TASK [default-telemetry : Get current time] ************************************
2026-02-02 13:54:37,084 p=36239 u=zuul n=ansible | Monday 02 February 2026  13:54:37 +0000 (0:02:00.318)       0:11:39.573 ******* 
2026-02-02 13:54:37,101 p=36239 u=zuul n=ansible | ok: [localhost]
2026-02-02 13:54:37,104 p=36239 u=zuul n=ansible | TASK [default-telemetry : Wait for new logs to be generated] *******************
2026-02-02 13:54:37,104 p=36239 u=zuul n=ansible | Monday 02 February 2026  13:54:37 +0000 (0:00:00.020)       0:11:39.593 ******* 
2026-02-02 13:56:37,373 p=36239 u=zuul n=ansible | ok: [localhost]
2026-02-02 13:56:37,378 p=36239 u=zuul n=ansible | TASK [default-telemetry : Get new telemetry-operator logs] *********************
2026-02-02 13:56:37,378 p=36239 u=zuul n=ansible | Monday 02 February 2026  13:56:37 +0000 (0:02:00.273)       0:13:39.867 ******* 
2026-02-02 13:56:37,794 p=36239 u=zuul n=ansible | changed: [localhost]
2026-02-02 13:56:37,799 p=36239 u=zuul n=ansible | TASK [default-telemetry : TEST Check that telemetry-operator logs don't include any errors when using the CustomMonitoringStack
RHOSO-123464] ***
2026-02-02 13:56:37,799 p=36239 u=zuul n=ansible | Monday 02 February 2026  13:56:37 +0000 (0:00:00.420)       0:13:40.287 ******* 
2026-02-02 13:56:37,819 p=36239 u=zuul n=ansible | ok: [localhost]
2026-02-02 13:56:37,823 p=36239 u=zuul n=ansible | TASK [default-telemetry : Patch Autoscaling to use a custom Prometheus instance] ***
2026-02-02 13:56:37,823 p=36239 u=zuul n=ansible | Monday 02 February 2026  13:56:37 +0000 (0:00:00.024)       0:13:40.312 ******* 
2026-02-02 13:56:38,407 p=36239 u=zuul n=ansible | changed: [localhost]
2026-02-02 13:56:38,412 p=36239 u=zuul n=ansible | TASK [default-telemetry : Wait until Autoscaling is ready] *********************
2026-02-02 13:56:38,412 p=36239 u=zuul n=ansible | Monday 02 February 2026  13:56:38 +0000 (0:00:00.589)       0:13:40.901 ******* 
2026-02-02 13:58:39,484 p=36239 u=zuul n=ansible | fatal: [localhost]: FAILED! => {"changed": true, "cmd": ["oc", "wait", "telemetry", "telemetry", "--for=condition=Ready", "--timeout=2m"], "delta": "0:02:00.853242", "end": "2026-02-02 13:58:39.457896", "msg": "non-zero return code", "rc": 1, "start": "2026-02-02 13:56:38.604654", "stderr": "error: timed out waiting for the condition on telemetries/telemetry", "stderr_lines": ["error: timed out waiting for the condition on telemetries/telemetry"], "stdout": "", "stdout_lines": []}
2026-02-02 13:58:39,484 p=36239 u=zuul n=ansible | ...ignoring
2026-02-02 13:58:39,488 p=36239 u=zuul n=ansible | TASK [default-telemetry : Get current time] ************************************
2026-02-02 13:58:39,489 p=36239 u=zuul n=ansible | Monday 02 February 2026  13:58:39 +0000 (0:02:01.076)       0:15:41.977 ******* 
2026-02-02 13:58:39,505 p=36239 u=zuul n=ansible | ok: [localhost]
2026-02-02 13:58:39,510 p=36239 u=zuul n=ansible | TASK [default-telemetry : Wait for new logs to be generated] *******************
2026-02-02 13:58:39,510 p=36239 u=zuul n=ansible | Monday 02 February 2026  13:58:39 +0000 (0:00:00.021)       0:15:41.999 ******* 
2026-02-02 14:00:39,800 p=36239 u=zuul n=ansible | ok: [localhost]
2026-02-02 14:00:39,804 p=36239 u=zuul n=ansible | TASK [default-telemetry : Get new telemetry-operator logs] *********************
2026-02-02 14:00:39,804 p=36239 u=zuul n=ansible | Monday 02 February 2026  14:00:39 +0000 (0:02:00.294)       0:17:42.293 ******* 
2026-02-02 14:00:40,191 p=36239 u=zuul n=ansible | changed: [localhost]
2026-02-02 14:00:40,196 p=36239 u=zuul n=ansible | TASK [default-telemetry : TEST Check that telemetry-operator logs don't include any errors when using a custom Prometheus instance for Autoscaling
RHOSO-123465] ***
2026-02-02 14:00:40,196 p=36239 u=zuul n=ansible | Monday 02 February 2026  14:00:40 +0000 (0:00:00.391)       0:17:42.685 ******* 
2026-02-02 14:00:40,213 p=36239 u=zuul n=ansible | ok: [localhost]
2026-02-02 14:00:40,217 p=36239 u=zuul n=ansible | TASK [default-telemetry : TEST Check Prometheus host is set correctly in the aodh-evaluator container when using a custom Prometheus
RHOSO-123466] ***
2026-02-02 14:00:40,217 p=36239 u=zuul n=ansible | Monday 02 February 2026  14:00:40 +0000 (0:00:00.020)       0:17:42.706 ******* 
2026-02-02 14:00:40,650 p=36239 u=zuul n=ansible | fatal: [localhost]: FAILED! => {"changed": true, "cmd": "oc rsh -c aodh-evaluator aodh-0 cat /etc/openstack/prometheus.yaml | grep host | cut -d \" \" -f 2", "delta": "0:00:00.161951", "end": "2026-02-02 14:00:40.611918", "failed_when_result": true, "msg": "", "rc": 0, "start": "2026-02-02 14:00:40.449967", "stderr": "Error from server (NotFound): pods \"aodh-0\" not found", "stderr_lines": ["Error from server (NotFound): pods \"aodh-0\" not found"], "stdout": "", "stdout_lines": []}
2026-02-02 14:00:40,650 p=36239 u=zuul n=ansible | ...ignoring
2026-02-02 14:00:40,655 p=36239 u=zuul n=ansible | TASK [default-telemetry : TEST Check Prometheus port is set correctly in the aodh-evaluator container when using a custom Prometheus
RHOSO-123467] ***
2026-02-02 14:00:40,655 p=36239 u=zuul n=ansible | Monday 02 February 2026  14:00:40 +0000 (0:00:00.437)       0:17:43.144 ******* 
2026-02-02 14:00:41,017 p=36239 u=zuul n=ansible | fatal: [localhost]: FAILED! => {"changed": true, "cmd": "oc rsh -c aodh-evaluator aodh-0 cat /etc/openstack/prometheus.yaml | grep port | cut -d \" \" -f 2", "delta": "0:00:00.143312", "end": "2026-02-02 14:00:40.988256", "failed_when_result": true, "msg": "", "rc": 0, "start": "2026-02-02 14:00:40.844944", "stderr": "Error from server (NotFound): pods \"aodh-0\" not found", "stderr_lines": ["Error from server (NotFound): pods \"aodh-0\" not found"], "stdout": "", "stdout_lines": []}
2026-02-02 14:00:41,017 p=36239 u=zuul n=ansible | ...ignoring
2026-02-02 14:00:41,021 p=36239 u=zuul n=ansible | TASK [default-telemetry : Test podlevel TLS with NAD for MetricStorage] ********
2026-02-02 14:00:41,021 p=36239 u=zuul n=ansible | Monday 02 February 2026  14:00:41 +0000 (0:00:00.366)       0:17:43.510 ******* 
2026-02-02 14:00:41,037 p=36239 u=zuul n=ansible | included: /home/zuul/src/github.com/openstack-k8s-operators/telemetry-operator/ci/default-telemetry/tasks/test_podlevel_tls_with_nad_metricstorage.yml for localhost
2026-02-02 14:00:41,080 p=36239 u=zuul n=ansible | TASK [default-telemetry : Enable TLS at a pod level] ***************************
2026-02-02 14:00:41,080 p=36239 u=zuul n=ansible | Monday 02 February 2026  14:00:41 +0000 (0:00:00.058)       0:17:43.569 ******* 
2026-02-02 14:00:41,586 p=36239 u=zuul n=ansible | changed: [localhost]
2026-02-02 14:00:41,591 p=36239 u=zuul n=ansible | TASK [default-telemetry : Set NAD for MetricStorage] ***************************
2026-02-02 14:00:41,591 p=36239 u=zuul n=ansible | Monday 02 February 2026  14:00:41 +0000 (0:00:00.511)       0:17:44.080 ******* 
2026-02-02 14:00:42,061 p=36239 u=zuul n=ansible | changed: [localhost]
2026-02-02 14:00:42,065 p=36239 u=zuul n=ansible | TASK [default-telemetry : Wait until MetricStorage is ready] *******************
2026-02-02 14:00:42,065 p=36239 u=zuul n=ansible | Monday 02 February 2026  14:00:42 +0000 (0:00:00.474)       0:17:44.554 ******* 
2026-02-02 14:02:42,391 p=36239 u=zuul n=ansible | fatal: [localhost]: FAILED! => {"changed": true, "cmd": ["oc", "wait", "telemetry", "telemetry", "--for=condition=Ready", "--timeout=2m"], "delta": "0:02:00.110880", "end": "2026-02-02 14:02:42.362280", "msg": "non-zero return code", "rc": 1, "start": "2026-02-02 14:00:42.251400", "stderr": "error: timed out waiting for the condition on telemetries/telemetry", "stderr_lines": ["error: timed out waiting for the condition on telemetries/telemetry"], "stdout": "", "stdout_lines": []}
2026-02-02 14:02:42,391 p=36239 u=zuul n=ansible | ...ignoring
2026-02-02 14:02:42,395 p=36239 u=zuul n=ansible | TASK [default-telemetry : Get current time] ************************************
2026-02-02 14:02:42,395 p=36239 u=zuul n=ansible | Monday 02 February 2026  14:02:42 +0000 (0:02:00.329)       0:19:44.884 ******* 
2026-02-02 14:02:42,412 p=36239 u=zuul n=ansible | ok: [localhost]
2026-02-02 14:02:42,416 p=36239 u=zuul n=ansible | TASK [default-telemetry : Wait for new logs to be generated] *******************
2026-02-02 14:02:42,417 p=36239 u=zuul n=ansible | Monday 02 February 2026  14:02:42 +0000 (0:00:00.021)       0:19:44.905 ******* 
2026-02-02 14:04:42,630 p=36239 u=zuul n=ansible | ok: [localhost]
2026-02-02 14:04:42,634 p=36239 u=zuul n=ansible | TASK [default-telemetry : Get new telemetry-operator logs] *********************
2026-02-02 14:04:42,635 p=36239 u=zuul n=ansible | Monday 02 February 2026  14:04:42 +0000 (0:02:00.218)       0:21:45.123 ******* 
2026-02-02 14:04:42,982 p=36239 u=zuul n=ansible | changed: [localhost]
2026-02-02 14:04:42,987 p=36239 u=zuul n=ansible | TASK [default-telemetry : TEST Check that telemetry-operator logs don't include any errors when using TLS at a pod level
OSPRH-14635] ***
2026-02-02 14:04:42,987 p=36239 u=zuul n=ansible | Monday 02 February 2026  14:04:42 +0000 (0:00:00.352)       0:21:45.476 ******* 
2026-02-02 14:04:43,010 p=36239 u=zuul n=ansible | ok: [localhost]
2026-02-02 14:04:43,015 p=36239 u=zuul n=ansible | TASK [default-telemetry : Test Prometheus_secret values] ***********************
2026-02-02 14:04:43,015 p=36239 u=zuul n=ansible | Monday 02 February 2026  14:04:43 +0000 (0:00:00.027)       0:21:45.503 ******* 
2026-02-02 14:04:43,030 p=36239 u=zuul n=ansible | included: /home/zuul/src/github.com/openstack-k8s-operators/telemetry-operator/ci/default-telemetry/tasks/test_prometheus_secret.yml for localhost
2026-02-02 14:04:43,070 p=36239 u=zuul n=ansible | TASK [default-telemetry : Get metric-storage-prometheus-endpoint secret values] ***
2026-02-02 14:04:43,071 p=36239 u=zuul n=ansible | Monday 02 February 2026  14:04:43 +0000 (0:00:00.055)       0:21:45.559 ******* 
2026-02-02 14:04:43,428 p=36239 u=zuul n=ansible | ok: [localhost]
2026-02-02 14:04:43,431 p=36239 u=zuul n=ansible | TASK [default-telemetry : Show secret decoded values] **************************
2026-02-02 14:04:43,431 p=36239 u=zuul n=ansible | Monday 02 February 2026  14:04:43 +0000 (0:00:00.360)       0:21:45.920 ******* 
2026-02-02 14:04:43,447 p=36239 u=zuul n=ansible | ok: [localhost] => {
    "secret_output.stdout_lines": [
        "ca_key=ca.crt",
        "ca_secret=ce**********vc",
        "host=metric-storage-prometheus.openstack.svc",
        "port=9090"
    ]
}
2026-02-02 14:04:43,450 p=36239 u=zuul n=ansible | TASK [default-telemetry : Assert secret contains expected values] **************
2026-02-02 14:04:43,450 p=36239 u=zuul n=ansible | Monday 02 February 2026  14:04:43 +0000 (0:00:00.018)       0:21:45.939 ******* 
2026-02-02 14:04:43,471 p=36239 u=zuul n=ansible | ok: [localhost] => {
    "changed": false,
    "msg": "Secret metric-storage-prometheus-endpoint contains all required values"
}
2026-02-02 14:04:43,483 p=36239 u=zuul n=ansible | PLAY RECAP *********************************************************************
2026-02-02 14:04:43,484 p=36239 u=zuul n=ansible | localhost                  : ok=71   changed=30   unreachable=0    failed=0    skipped=1    rescued=0    ignored=10  
2026-02-02 14:04:43,489 p=36239 u=zuul n=ansible | Monday 02 February 2026  14:04:43 +0000 (0:00:00.039)       0:21:45.978 ******* 
2026-02-02 14:04:43,490 p=36239 u=zuul n=ansible | =============================================================================== 
2026-02-02 14:04:43,490 p=36239 u=zuul n=ansible | default-telemetry : Wait until Autoscaling is ready ------------------- 121.08s
2026-02-02 14:04:43,490 p=36239 u=zuul n=ansible | default-telemetry : Wait until MetricStorage is ready ----------------- 120.33s
2026-02-02 14:04:43,490 p=36239 u=zuul n=ansible | default-telemetry : Wait for new logs to be generated ----------------- 120.33s
2026-02-02 14:04:43,490 p=36239 u=zuul n=ansible | default-telemetry : Wait until MetricStorage is ready ----------------- 120.32s
2026-02-02 14:04:43,490 p=36239 u=zuul n=ansible | default-telemetry : Wait for new logs to be generated ----------------- 120.31s
2026-02-02 14:04:43,490 p=36239 u=zuul n=ansible | default-telemetry : Wait until reconciliation finishes ---------------- 120.30s
2026-02-02 14:04:43,490 p=36239 u=zuul n=ansible | default-telemetry : Wait for new logs to be generated ----------------- 120.29s
2026-02-02 14:04:43,490 p=36239 u=zuul n=ansible | default-telemetry : Wait for new logs to be generated ----------------- 120.27s
2026-02-02 14:04:43,490 p=36239 u=zuul n=ansible | default-telemetry : Wait for new logs to be generated ----------------- 120.27s
2026-02-02 14:04:43,490 p=36239 u=zuul n=ansible | default-telemetry : Wait for new logs to be generated ----------------- 120.22s
2026-02-02 14:04:43,490 p=36239 u=zuul n=ansible | default-telemetry : Wait until Autoscaling and MetricStorage are ready -- 41.83s
2026-02-02 14:04:43,490 p=36239 u=zuul n=ansible | default-telemetry : Wait for the required resource to be created ------- 21.06s
2026-02-02 14:04:43,490 p=36239 u=zuul n=ansible | default-telemetry : Wait for the resources to be available ------------- 20.15s
2026-02-02 14:04:43,490 p=36239 u=zuul n=ansible | default-telemetry : Get Autoscaling condition values -------------------- 5.10s
2026-02-02 14:04:43,490 p=36239 u=zuul n=ansible | default-telemetry : Get MetricStorage condition values ------------------ 3.38s
2026-02-02 14:04:43,490 p=36239 u=zuul n=ansible | default-telemetry : Patch Autoscaling to use a custom Prometheus instance --- 0.59s
2026-02-02 14:04:43,490 p=36239 u=zuul n=ansible | default-telemetry : Patch MetricStorage to use CustomMonitoringStack field --- 0.59s
2026-02-02 14:04:43,490 p=36239 u=zuul n=ansible | default-telemetry : Wait until telemetry is finished reconciling and its conditions are all True --- 0.52s
2026-02-02 14:04:43,490 p=36239 u=zuul n=ansible | default-telemetry : Enable TLS at a pod level --------------------------- 0.51s
2026-02-02 14:04:43,490 p=36239 u=zuul n=ansible | default-telemetry : Enable Autoscaling ---------------------------------- 0.50s
2026-02-02 14:04:43,490 p=36239 u=zuul n=ansible | default-telemetry : Enable MetricStorage -------------------------------- 0.48s
2026-02-02 14:04:43,490 p=36239 u=zuul n=ansible | default-telemetry : Set NAD for MetricStorage --------------------------- 0.47s
2026-02-02 14:04:43,490 p=36239 u=zuul n=ansible | default-telemetry : TEST Check Prometheus host is set correctly in the aodh-evaluator container when using a custom Prometheus
RHOSO-123466
 --- 0.44s
2026-02-02 14:04:43,490 p=36239 u=zuul n=ansible | default-telemetry : Get new telemetry-operator logs --------------------- 0.42s
2026-02-02 14:04:43,490 p=36239 u=zuul n=ansible | default-telemetry : Get new telemetry-operator logs --------------------- 0.42s
2026-02-02 14:04:43,490 p=36239 u=zuul n=ansible | default-telemetry : Get new telemetry-operator logs --------------------- 0.39s
2026-02-02 14:04:43,490 p=36239 u=zuul n=ansible | default-telemetry : Get new telemetry-operator logs --------------------- 0.38s
2026-02-02 14:04:43,490 p=36239 u=zuul n=ansible | default-telemetry : Get new telemetry-operator logs --------------------- 0.37s
2026-02-02 14:04:43,490 p=36239 u=zuul n=ansible | default-telemetry : TEST Check Prometheus port is set correctly in the aodh-evaluator container when using a custom Prometheus
RHOSO-123467
 --- 0.37s
2026-02-02 14:04:43,490 p=36239 u=zuul n=ansible | default-telemetry : Get metric-storage-prometheus-endpoint secret values --- 0.36s
2026-02-02 14:04:43,490 p=36239 u=zuul n=ansible | default-telemetry : Get telemetry-operator pod data --------------------- 0.36s
2026-02-02 14:04:43,490 p=36239 u=zuul n=ansible | default-telemetry : Get new telemetry-operator logs --------------------- 0.35s
2026-02-02 14:04:43,490 p=36239 u=zuul n=ansible | default-telemetry : Get MetricStorage condition types ------------------- 0.35s
2026-02-02 14:04:43,490 p=36239 u=zuul n=ansible | default-telemetry : Get telemetry-operator failed container logs -------- 0.34s
2026-02-02 14:04:43,490 p=36239 u=zuul n=ansible | default-telemetry : Get Autoscaling condition types --------------------- 0.34s
2026-02-02 14:04:43,491 p=36239 u=zuul n=ansible | default-telemetry : Get telemetry-operator failed container logs -------- 0.34s
2026-02-02 14:04:43,491 p=36239 u=zuul n=ansible | default-telemetry : Get telemetry-operator pod data --------------------- 0.34s
2026-02-02 14:04:43,491 p=36239 u=zuul n=ansible | default-telemetry : Create the COO subscription ------------------------- 0.33s
2026-02-02 14:04:43,491 p=36239 u=zuul n=ansible | default-telemetry : Construct Autoscaling condition dictionary ---------- 0.15s
2026-02-02 14:04:43,491 p=36239 u=zuul n=ansible | default-telemetry : Construct MetricStorage condition dictionary -------- 0.11s
2026-02-02 14:04:43,491 p=36239 u=zuul n=ansible | default-telemetry : Test podlevel TLS with NAD for MetricStorage -------- 0.06s
2026-02-02 14:04:43,491 p=36239 u=zuul n=ansible | default-telemetry : Test Prometheus_secret values ----------------------- 0.06s
2026-02-02 14:04:43,491 p=36239 u=zuul n=ansible | default-telemetry : Test CustomMonitoringStack -------------------------- 0.05s
2026-02-02 14:04:43,491 p=36239 u=zuul n=ansible | default-telemetry : Check telemetry without COO and with Autoscaling and MetricStorage enabled --- 0.04s
2026-02-02 14:04:43,491 p=36239 u=zuul n=ansible | default-telemetry : Assert secret contains expected values -------------- 0.04s
2026-02-02 14:04:43,491 p=36239 u=zuul n=ansible | default-telemetry : Check telemetry with COO and with Autoscaling and MetricStorage enabled --- 0.04s
2026-02-02 14:04:43,491 p=36239 u=zuul n=ansible | default-telemetry : TEST Check that all MetricStorage conditions are as expected when COO isn't installed
RHOSO-123461
 --- 0.04s
2026-02-02 14:04:43,491 p=36239 u=zuul n=ansible | default-telemetry : Enable Autoscaling and MetricStorage ---------------- 0.04s
2026-02-02 14:04:43,491 p=36239 u=zuul n=ansible | default-telemetry : TEST Check that telemetry-operator logs don't include any errors when using TLS at a pod level
OSPRH-14635
 --- 0.03s
2026-02-02 14:04:43,491 p=36239 u=zuul n=ansible | default-telemetry : Check telemetry without COO and without Autoscaling and MetricStorage enabled --- 0.03s
2026-02-02 14:04:43,491 p=36239 u=zuul n=ansible | default-telemetry : TEST Check that telemetry-operator logs include expected error logs when MetricStorage is enabled, but COO isn't installed
RHOSO-123459
 --- 0.03s
2026-02-02 14:04:43,491 p=36239 u=zuul n=ansible | default-telemetry : TEST Check telemetry-operator logs for errors after it's started
RHOSO-123456
 --- 0.03s
2026-02-02 14:04:43,491 p=36239 u=zuul n=ansible | default-telemetry : TEST Check that all Autoscaling conditions are as expected when COO isn't installed
RHOSO-123462
 --- 0.02s
2026-02-02 14:04:43,491 p=36239 u=zuul n=ansible | default-telemetry : Output logs of failed container for debugging purposes --- 0.02s
2026-02-02 14:04:43,491 p=36239 u=zuul n=ansible | default-telemetry : TEST Check that telemetry-operator logs don't include any errors when using the CustomMonitoringStack
RHOSO-123464
 --- 0.02s
2026-02-02 14:04:43,491 p=36239 u=zuul n=ansible | default-telemetry : Enable Autoscaling and MetricStorage ---------------- 0.02s
2026-02-02 14:04:43,491 p=36239 u=zuul n=ansible | default-telemetry : TEST Check that telemetry-operator logs don't include errors when MetricStorage is enabled, but COO isn't installed
RHOSO-123460
 --- 0.02s
2026-02-02 14:04:43,491 p=36239 u=zuul n=ansible | default-telemetry : Get telemetry-operator restart counts --------------- 0.02s
2026-02-02 14:04:43,491 p=36239 u=zuul n=ansible | default-telemetry : TEST Check that telemetry-operator logs don't include any errors after COO is installed
RHOSO-123463
 --- 0.02s
2026-02-02 14:04:43,491 p=36239 u=zuul n=ansible | default-telemetry : Output logs of failed container for debugging purposes --- 0.02s
2026-02-02 14:04:43,491 p=36239 u=zuul n=ansible | default-telemetry : Get current time ------------------------------------ 0.02s
2026-02-02 14:04:43,491 p=36239 u=zuul n=ansible | default-telemetry : Get current time ------------------------------------ 0.02s
2026-02-02 14:04:43,491 p=36239 u=zuul n=ansible | default-telemetry : Get current time ------------------------------------ 0.02s
2026-02-02 14:04:43,492 p=36239 u=zuul n=ansible | default-telemetry : Get current time ------------------------------------ 0.02s
2026-02-02 14:04:43,492 p=36239 u=zuul n=ansible | default-telemetry : TEST Check that telemetry-operator logs don't include any errors when using a custom Prometheus instance for Autoscaling
RHOSO-123465
 --- 0.02s
2026-02-02 14:04:43,492 p=36239 u=zuul n=ansible | default-telemetry : Get current time ------------------------------------ 0.02s
2026-02-02 14:04:43,492 p=36239 u=zuul n=ansible | default-telemetry : Get telemetry-operator restart counts --------------- 0.02s
2026-02-02 14:04:43,492 p=36239 u=zuul n=ansible | default-telemetry : Get telemetry-operator container statuses ----------- 0.02s
2026-02-02 14:04:43,492 p=36239 u=zuul n=ansible | default-telemetry : Show secret decoded values -------------------------- 0.02s
2026-02-02 14:04:43,492 p=36239 u=zuul n=ansible | default-telemetry : Get current time ------------------------------------ 0.02s
2026-02-02 14:04:43,492 p=36239 u=zuul n=ansible | default-telemetry : Output found errors for debugging purposes ---------- 0.02s