LAST SEEN TYPE REASON OBJECT MESSAGE 89m Normal RequirementsUnknown clusterserviceversion/cluster-logging.v6.2.9 requirements not yet checked 89m Normal RequirementsNotMet clusterserviceversion/cluster-logging.v6.2.9 one or more requirements couldn't be found 89m Normal SuccessfulCreate replicaset/cluster-logging-operator-66689c4bbf Created pod: cluster-logging-operator-66689c4bbf-f7fr7 89m Normal Scheduled pod/cluster-logging-operator-66689c4bbf-f7fr7 Successfully assigned openshift-logging/cluster-logging-operator-66689c4bbf-f7fr7 to crc 89m Normal AllRequirementsMet clusterserviceversion/cluster-logging.v6.2.9 all requirements found, attempting install 89m Normal ScalingReplicaSet deployment/cluster-logging-operator Scaled up replica set cluster-logging-operator-66689c4bbf to 1 89m Normal InstallWaiting clusterserviceversion/cluster-logging.v6.2.9 installing: waiting for deployment cluster-logging-operator to become ready: deployment "cluster-logging-operator" not available: Deployment does not have minimum availability. 89m Normal InstallSucceeded clusterserviceversion/cluster-logging.v6.2.9 waiting for install components to report healthy 89m Normal Pulling pod/cluster-logging-operator-66689c4bbf-f7fr7 Pulling image "registry.redhat.io/openshift-logging/cluster-logging-rhel9-operator@sha256:bf86702a43f0ab4fda541a3aa8ace4560088be90be5eb59cd475ce1d97f5afef" 89m Normal AddedInterface pod/cluster-logging-operator-66689c4bbf-f7fr7 Add eth0 [10.217.0.51/23] from ovn-kubernetes 89m Normal Pulled pod/cluster-logging-operator-66689c4bbf-f7fr7 Successfully pulled image "registry.redhat.io/openshift-logging/cluster-logging-rhel9-operator@sha256:bf86702a43f0ab4fda541a3aa8ace4560088be90be5eb59cd475ce1d97f5afef" in 5.349s (5.349s including waiting). Image size: 343887794 bytes. 89m Normal Started pod/cluster-logging-operator-66689c4bbf-f7fr7 Started container cluster-logging-operator 89m Normal Created pod/cluster-logging-operator-66689c4bbf-f7fr7 Created container cluster-logging-operator 89m Normal InstallSucceeded clusterserviceversion/cluster-logging.v6.2.9 install strategy completed with no errors 89m Normal Scheduled pod/logging-loki-distributor-9c6b6d984-h7xw6 Successfully assigned openshift-logging/logging-loki-distributor-9c6b6d984-h7xw6 to crc 89m Normal ScalingReplicaSet deployment/logging-loki-distributor Scaled up replica set logging-loki-distributor-9c6b6d984 to 1 89m Normal SuccessfulCreate replicaset/logging-loki-distributor-9c6b6d984 Created pod: logging-loki-distributor-9c6b6d984-h7xw6 89m Normal Pulling pod/logging-loki-distributor-9c6b6d984-h7xw6 Pulling image "registry.redhat.io/openshift-logging/logging-loki-rhel9@sha256:ebf28f8c032032395ba9a7aa247f68a25e9484857588e375b4660bc04aa0e0fb" 89m Normal SuccessfulCreate statefulset/logging-loki-index-gateway create Pod logging-loki-index-gateway-0 in StatefulSet logging-loki-index-gateway successful 89m Normal Scheduled pod/logging-loki-query-frontend-ff66c4dc9-5d4kx Successfully assigned openshift-logging/logging-loki-query-frontend-ff66c4dc9-5d4kx to crc 89m Normal ScalingReplicaSet deployment/logging-loki-querier Scaled up replica set logging-loki-querier-6dcbdf8bb8 to 1 89m Normal SuccessfulCreate replicaset/logging-loki-querier-6dcbdf8bb8 Created pod: logging-loki-querier-6dcbdf8bb8-rbp8f 89m Normal WaitForPodScheduled persistentvolumeclaim/storage-logging-loki-compactor-0 waiting for pod logging-loki-compactor-0 to be scheduled 89m Normal Scheduled pod/logging-loki-gateway-54c568c9c8-mwlnd Successfully assigned openshift-logging/logging-loki-gateway-54c568c9c8-mwlnd to crc 89m Normal ProvisioningSucceeded persistentvolumeclaim/wal-logging-loki-ingester-0 Successfully provisioned volume pvc-72fdc5da-cef5-4245-a008-b32a01be29af 89m Normal AddedInterface pod/logging-loki-querier-6dcbdf8bb8-rbp8f Add eth0 [10.217.0.54/23] from ovn-kubernetes 89m Normal ExternalProvisioning persistentvolumeclaim/storage-logging-loki-index-gateway-0 Waiting for a volume to be created either by the external provisioner 'kubevirt.io.hostpath-provisioner' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. 89m Normal WaitForFirstConsumer persistentvolumeclaim/storage-logging-loki-index-gateway-0 waiting for first consumer to be created before binding 89m Normal ProvisioningSucceeded persistentvolumeclaim/storage-logging-loki-compactor-0 Successfully provisioned volume pvc-12096e3b-ceeb-4544-bc52-ffd2233bef18 89m Normal ScalingReplicaSet deployment/logging-loki-query-frontend Scaled up replica set logging-loki-query-frontend-ff66c4dc9 to 1 89m Normal Scheduled pod/logging-loki-querier-6dcbdf8bb8-rbp8f Successfully assigned openshift-logging/logging-loki-querier-6dcbdf8bb8-rbp8f to crc 89m Normal SuccessfulCreate statefulset/logging-loki-ingester create Pod logging-loki-ingester-0 in StatefulSet logging-loki-ingester successful 89m Normal SuccessfulCreate statefulset/logging-loki-ingester create Claim storage-logging-loki-ingester-0 Pod logging-loki-ingester-0 in StatefulSet logging-loki-ingester success 89m Normal NoPods poddisruptionbudget/logging-loki-ingester No matching pods found 89m Normal SuccessfulCreate statefulset/logging-loki-ingester create Claim wal-logging-loki-ingester-0 Pod logging-loki-ingester-0 in StatefulSet logging-loki-ingester success 89m Normal ExternalProvisioning persistentvolumeclaim/storage-logging-loki-compactor-0 Waiting for a volume to be created either by the external provisioner 'kubevirt.io.hostpath-provisioner' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. 89m Normal SuccessfulCreate statefulset/logging-loki-compactor create Claim storage-logging-loki-compactor-0 Pod logging-loki-compactor-0 in StatefulSet logging-loki-compactor success 89m Normal SuccessfulCreate statefulset/logging-loki-compactor create Pod logging-loki-compactor-0 in StatefulSet logging-loki-compactor successful 89m Normal Provisioning persistentvolumeclaim/storage-logging-loki-index-gateway-0 External provisioner is provisioning volume for claim "openshift-logging/storage-logging-loki-index-gateway-0" 89m Normal AddedInterface pod/logging-loki-distributor-9c6b6d984-h7xw6 Add eth0 [10.217.0.53/23] from ovn-kubernetes 89m Normal ProvisioningSucceeded persistentvolumeclaim/storage-logging-loki-index-gateway-0 Successfully provisioned volume pvc-8a81539e-94e4-4598-90f1-0f179be2eaeb 89m Normal WaitForFirstConsumer persistentvolumeclaim/storage-logging-loki-ingester-0 waiting for first consumer to be created before binding 89m Normal Provisioning persistentvolumeclaim/storage-logging-loki-ingester-0 External provisioner is provisioning volume for claim "openshift-logging/storage-logging-loki-ingester-0" 89m Normal ExternalProvisioning persistentvolumeclaim/storage-logging-loki-ingester-0 Waiting for a volume to be created either by the external provisioner 'kubevirt.io.hostpath-provisioner' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. 89m Normal ProvisioningSucceeded persistentvolumeclaim/storage-logging-loki-ingester-0 Successfully provisioned volume pvc-d86f0c19-27d0-43bf-8b7d-e69b3fe13397 89m Normal Provisioning persistentvolumeclaim/wal-logging-loki-ingester-0 External provisioner is provisioning volume for claim "openshift-logging/wal-logging-loki-ingester-0" 89m Normal ExternalProvisioning persistentvolumeclaim/wal-logging-loki-ingester-0 Waiting for a volume to be created either by the external provisioner 'kubevirt.io.hostpath-provisioner' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. 89m Normal WaitForFirstConsumer persistentvolumeclaim/wal-logging-loki-ingester-0 waiting for first consumer to be created before binding 89m Normal Provisioning persistentvolumeclaim/storage-logging-loki-compactor-0 External provisioner is provisioning volume for claim "openshift-logging/storage-logging-loki-compactor-0" 89m Normal SuccessfulCreate replicaset/logging-loki-gateway-54c568c9c8 Created pod: logging-loki-gateway-54c568c9c8-mwlnd 89m Normal SuccessfulCreate replicaset/logging-loki-gateway-54c568c9c8 Created pod: logging-loki-gateway-54c568c9c8-98zs2 89m Normal SuccessfulCreate replicaset/logging-loki-query-frontend-ff66c4dc9 Created pod: logging-loki-query-frontend-ff66c4dc9-5d4kx 89m Normal Scheduled pod/logging-loki-gateway-54c568c9c8-98zs2 Successfully assigned openshift-logging/logging-loki-gateway-54c568c9c8-98zs2 to crc 89m Normal ScalingReplicaSet deployment/logging-loki-gateway Scaled up replica set logging-loki-gateway-54c568c9c8 to 2 89m Normal NoPods poddisruptionbudget/logging-loki-index-gateway No matching pods found 89m Normal SuccessfulCreate statefulset/logging-loki-index-gateway create Claim storage-logging-loki-index-gateway-0 Pod logging-loki-index-gateway-0 in StatefulSet logging-loki-index-gateway success 89m Normal Pulling pod/logging-loki-ingester-0 Pulling image "registry.redhat.io/openshift-logging/logging-loki-rhel9@sha256:ebf28f8c032032395ba9a7aa247f68a25e9484857588e375b4660bc04aa0e0fb" 89m Normal Pulling pod/logging-loki-querier-6dcbdf8bb8-rbp8f Pulling image "registry.redhat.io/openshift-logging/logging-loki-rhel9@sha256:ebf28f8c032032395ba9a7aa247f68a25e9484857588e375b4660bc04aa0e0fb" 89m Normal AddedInterface pod/logging-loki-ingester-0 Add eth0 [10.217.0.58/23] from ovn-kubernetes 89m Normal Scheduled pod/logging-loki-compactor-0 Successfully assigned openshift-logging/logging-loki-compactor-0 to crc 89m Normal Pulling pod/logging-loki-query-frontend-ff66c4dc9-5d4kx Pulling image "registry.redhat.io/openshift-logging/logging-loki-rhel9@sha256:ebf28f8c032032395ba9a7aa247f68a25e9484857588e375b4660bc04aa0e0fb" 89m Normal Scheduled pod/logging-loki-index-gateway-0 Successfully assigned openshift-logging/logging-loki-index-gateway-0 to crc 89m Normal AddedInterface pod/logging-loki-gateway-54c568c9c8-98zs2 Add eth0 [10.217.0.57/23] from ovn-kubernetes 89m Normal Pulling pod/logging-loki-gateway-54c568c9c8-mwlnd Pulling image "registry.redhat.io/openshift-logging/lokistack-gateway-rhel9@sha256:f4f8fc520eae06a5c7d8b98ef21bad88253666fc27db9b6bf35b19a63173a167" 89m Normal AddedInterface pod/logging-loki-query-frontend-ff66c4dc9-5d4kx Add eth0 [10.217.0.55/23] from ovn-kubernetes 89m Normal Scheduled pod/logging-loki-ingester-0 Successfully assigned openshift-logging/logging-loki-ingester-0 to crc 89m Normal AddedInterface pod/logging-loki-gateway-54c568c9c8-mwlnd Add eth0 [10.217.0.56/23] from ovn-kubernetes 89m Normal Pulling pod/logging-loki-gateway-54c568c9c8-98zs2 Pulling image "registry.redhat.io/openshift-logging/lokistack-gateway-rhel9@sha256:f4f8fc520eae06a5c7d8b98ef21bad88253666fc27db9b6bf35b19a63173a167" 89m Normal AddedInterface pod/logging-loki-index-gateway-0 Add eth0 [10.217.0.61/23] from ovn-kubernetes 89m Normal Pulling pod/logging-loki-compactor-0 Pulling image "registry.redhat.io/openshift-logging/logging-loki-rhel9@sha256:ebf28f8c032032395ba9a7aa247f68a25e9484857588e375b4660bc04aa0e0fb" 89m Normal AddedInterface pod/logging-loki-compactor-0 Add eth0 [10.217.0.60/23] from ovn-kubernetes 89m Normal Pulling pod/logging-loki-index-gateway-0 Pulling image "registry.redhat.io/openshift-logging/logging-loki-rhel9@sha256:ebf28f8c032032395ba9a7aa247f68a25e9484857588e375b4660bc04aa0e0fb" 89m Normal Pulled pod/logging-loki-query-frontend-ff66c4dc9-5d4kx Successfully pulled image "registry.redhat.io/openshift-logging/logging-loki-rhel9@sha256:ebf28f8c032032395ba9a7aa247f68a25e9484857588e375b4660bc04aa0e0fb" in 11.73s (11.73s including waiting). Image size: 225648085 bytes. 89m Normal Pulled pod/logging-loki-distributor-9c6b6d984-h7xw6 Successfully pulled image "registry.redhat.io/openshift-logging/logging-loki-rhel9@sha256:ebf28f8c032032395ba9a7aa247f68a25e9484857588e375b4660bc04aa0e0fb" in 11.884s (11.884s including waiting). Image size: 225648085 bytes. 89m Normal Pulled pod/logging-loki-gateway-54c568c9c8-98zs2 Successfully pulled image "registry.redhat.io/openshift-logging/lokistack-gateway-rhel9@sha256:f4f8fc520eae06a5c7d8b98ef21bad88253666fc27db9b6bf35b19a63173a167" in 10.863s (10.863s including waiting). Image size: 181406084 bytes. 89m Normal Pulled pod/logging-loki-ingester-0 Successfully pulled image "registry.redhat.io/openshift-logging/logging-loki-rhel9@sha256:ebf28f8c032032395ba9a7aa247f68a25e9484857588e375b4660bc04aa0e0fb" in 10.918s (10.918s including waiting). Image size: 225648085 bytes. 89m Normal Pulled pod/logging-loki-compactor-0 Successfully pulled image "registry.redhat.io/openshift-logging/logging-loki-rhel9@sha256:ebf28f8c032032395ba9a7aa247f68a25e9484857588e375b4660bc04aa0e0fb" in 10.736s (10.736s including waiting). Image size: 225648085 bytes. 89m Normal Pulled pod/logging-loki-index-gateway-0 Successfully pulled image "registry.redhat.io/openshift-logging/logging-loki-rhel9@sha256:ebf28f8c032032395ba9a7aa247f68a25e9484857588e375b4660bc04aa0e0fb" in 10.608s (10.608s including waiting). Image size: 225648085 bytes. 89m Normal Pulled pod/logging-loki-querier-6dcbdf8bb8-rbp8f Successfully pulled image "registry.redhat.io/openshift-logging/logging-loki-rhel9@sha256:ebf28f8c032032395ba9a7aa247f68a25e9484857588e375b4660bc04aa0e0fb" in 11.209s (11.209s including waiting). Image size: 225648085 bytes. 89m Normal Pulled pod/logging-loki-gateway-54c568c9c8-mwlnd Successfully pulled image "registry.redhat.io/openshift-logging/lokistack-gateway-rhel9@sha256:f4f8fc520eae06a5c7d8b98ef21bad88253666fc27db9b6bf35b19a63173a167" in 11.313s (11.313s including waiting). Image size: 181406084 bytes. 89m Normal Created pod/logging-loki-gateway-54c568c9c8-98zs2 Created container gateway 89m Normal Started pod/logging-loki-querier-6dcbdf8bb8-rbp8f Started container loki-querier 89m Normal Created pod/logging-loki-querier-6dcbdf8bb8-rbp8f Created container loki-querier 89m Normal Started pod/logging-loki-index-gateway-0 Started container loki-index-gateway 89m Normal Pulling pod/logging-loki-gateway-54c568c9c8-mwlnd Pulling image "registry.redhat.io/openshift-logging/opa-openshift-rhel9@sha256:d385c61e4b289c60cbb712e7c85c4af518e680cf24e9f7fb10acae2b281f08e8" 89m Normal Created pod/logging-loki-index-gateway-0 Created container loki-index-gateway 89m Normal Pulling pod/logging-loki-gateway-54c568c9c8-98zs2 Pulling image "registry.redhat.io/openshift-logging/opa-openshift-rhel9@sha256:d385c61e4b289c60cbb712e7c85c4af518e680cf24e9f7fb10acae2b281f08e8" 89m Normal Started pod/logging-loki-gateway-54c568c9c8-98zs2 Started container gateway 89m Normal Created pod/logging-loki-gateway-54c568c9c8-mwlnd Created container gateway 89m Normal Started pod/logging-loki-ingester-0 Started container loki-ingester 89m Normal Created pod/logging-loki-ingester-0 Created container loki-ingester 89m Normal Started pod/logging-loki-gateway-54c568c9c8-mwlnd Started container gateway 89m Normal Started pod/logging-loki-query-frontend-ff66c4dc9-5d4kx Started container loki-query-frontend 89m Normal Created pod/logging-loki-query-frontend-ff66c4dc9-5d4kx Created container loki-query-frontend 89m Normal Created pod/logging-loki-distributor-9c6b6d984-h7xw6 Created container loki-distributor 89m Normal Started pod/logging-loki-distributor-9c6b6d984-h7xw6 Started container loki-distributor 89m Normal Started pod/logging-loki-compactor-0 Started container loki-compactor 89m Normal Created pod/logging-loki-compactor-0 Created container loki-compactor 89m Normal Pulled pod/logging-loki-gateway-54c568c9c8-98zs2 Successfully pulled image "registry.redhat.io/openshift-logging/opa-openshift-rhel9@sha256:d385c61e4b289c60cbb712e7c85c4af518e680cf24e9f7fb10acae2b281f08e8" in 2.275s (2.275s including waiting). Image size: 161395402 bytes. 89m Normal Pulled pod/logging-loki-gateway-54c568c9c8-mwlnd Successfully pulled image "registry.redhat.io/openshift-logging/opa-openshift-rhel9@sha256:d385c61e4b289c60cbb712e7c85c4af518e680cf24e9f7fb10acae2b281f08e8" in 2.279s (2.279s including waiting). Image size: 161395402 bytes. 89m Normal Started pod/logging-loki-gateway-54c568c9c8-mwlnd Started container opa 89m Normal Created pod/logging-loki-gateway-54c568c9c8-mwlnd Created container opa 89m Normal Started pod/logging-loki-gateway-54c568c9c8-98zs2 Started container opa 89m Normal Created pod/logging-loki-gateway-54c568c9c8-98zs2 Created container opa 88m Warning ProbeError pod/logging-loki-ingester-0 Readiness probe error: HTTP probe failed with statuscode: 503... 88m Warning ProbeError pod/logging-loki-ingester-0 Readiness probe error: HTTP probe failed with statuscode: 503... 88m Warning Unhealthy pod/logging-loki-ingester-0 Readiness probe failed: HTTP probe failed with statuscode: 503 88m Warning FailedMount pod/collector-dcnzv MountVolume.SetUp failed for volume "collector-syslog-receiver" : secret "collector-syslog-receiver" not found 88m Normal SuccessfulDelete daemonset/collector Deleted pod: collector-dcnzv 88m Normal SuccessfulCreate daemonset/collector Created pod: collector-dcnzv 88m Normal Scheduled pod/collector-dcnzv Successfully assigned openshift-logging/collector-dcnzv to crc 88m Warning FailedMount pod/collector-dcnzv MountVolume.SetUp failed for volume "metrics" : secret "collector-metrics" not found 87m Normal SuccessfulCreate daemonset/collector Created pod: collector-7q2cm 87m Normal Scheduled pod/collector-7q2cm Successfully assigned openshift-logging/collector-7q2cm to crc 87m Normal AddedInterface pod/collector-7q2cm Add eth0 [10.217.0.66/23] from ovn-kubernetes 87m Normal Pulling pod/collector-7q2cm Pulling image "registry.redhat.io/openshift-logging/vector-rhel9@sha256:bacedab86fc0ed94a09cfa903e4d88e85b321df18224c43efcba4f6640aa8203" 87m Normal Pulled pod/collector-7q2cm Successfully pulled image "registry.redhat.io/openshift-logging/vector-rhel9@sha256:bacedab86fc0ed94a09cfa903e4d88e85b321df18224c43efcba4f6640aa8203" in 5.17s (5.17s including waiting). Image size: 263139944 bytes. 87m Normal Started pod/collector-7q2cm Started container collector 87m Normal Created pod/collector-7q2cm Created container collector 82m Normal nodeAssigned service/openstack-logging announcing from node "crc" with protocol "layer2" 82m Warning deprecatedAnnotation service/openstack-logging Service uses deprecated annotation metallb.universe.tf/loadBalancerIPs 82m Warning deprecatedAnnotation service/openstack-logging Service uses deprecated annotation metallb.universe.tf/allow-shared-ip 82m Normal IPAllocated service/openstack-logging Assigned IP ["172.17.0.80"] 82m Warning deprecatedAnnotation service/openstack-logging Service uses deprecated annotation metallb.universe.tf/address-pool 54m Warning Unhealthy pod/logging-loki-ingester-0 Readiness probe failed: Get "https://10.217.0.58:3101/ready": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) 54m Warning ProbeError pod/logging-loki-ingester-0 Readiness probe error: Get "https://10.217.0.58:3101/ready": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)... 46m Warning ProbeError pod/logging-loki-gateway-54c568c9c8-98zs2 Liveness probe error: HTTP probe failed with statuscode: 503... 46m Warning Unhealthy pod/logging-loki-gateway-54c568c9c8-98zs2 Liveness probe failed: HTTP probe failed with statuscode: 503 43m Warning ProbeError pod/logging-loki-gateway-54c568c9c8-98zs2 Readiness probe error: Get "https://10.217.0.57:8081/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 43m Warning Unhealthy pod/logging-loki-gateway-54c568c9c8-98zs2 Readiness probe failed: Get "https://10.217.0.57:8081/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 39m Warning ProbeError pod/logging-loki-gateway-54c568c9c8-mwlnd Readiness probe error: Get "https://10.217.0.56:8083/ready": context deadline exceeded... 39m Warning Unhealthy pod/logging-loki-gateway-54c568c9c8-mwlnd Readiness probe failed: Get "https://10.217.0.56:8083/ready": context deadline exceeded 20m Warning Unhealthy pod/logging-loki-query-frontend-ff66c4dc9-5d4kx Readiness probe failed: Get "https://10.217.0.55:3101/loki/api/v1/status/buildinfo": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) 20m Warning ProbeError pod/logging-loki-query-frontend-ff66c4dc9-5d4kx Readiness probe error: Get "https://10.217.0.55:3101/loki/api/v1/status/buildinfo": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)... 20m Warning Unhealthy pod/logging-loki-gateway-54c568c9c8-98zs2 Readiness probe failed: Get "https://10.217.0.57:8081/ready": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) 20m Warning ProbeError pod/logging-loki-gateway-54c568c9c8-98zs2 Readiness probe error: Get "https://10.217.0.57:8081/ready": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)... 20m Warning ProbeError pod/logging-loki-distributor-9c6b6d984-h7xw6 Liveness probe error: Get "https://10.217.0.53:3101/loki/api/v1/status/buildinfo": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)... 20m Warning Unhealthy pod/logging-loki-distributor-9c6b6d984-h7xw6 Liveness probe failed: Get "https://10.217.0.53:3101/loki/api/v1/status/buildinfo": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) 20m Warning ProbeError pod/logging-loki-querier-6dcbdf8bb8-rbp8f Readiness probe error: Get "https://10.217.0.54:3101/ready": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)... 20m Warning ProbeError pod/logging-loki-gateway-54c568c9c8-mwlnd Readiness probe error: Get "https://10.217.0.56:8081/ready": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)... 20m Warning Unhealthy pod/logging-loki-gateway-54c568c9c8-mwlnd Readiness probe failed: Get "https://10.217.0.56:8081/ready": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) 20m Warning Unhealthy pod/logging-loki-querier-6dcbdf8bb8-rbp8f Readiness probe failed: Get "https://10.217.0.54:3101/ready": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) 20m Warning ProbeError pod/logging-loki-gateway-54c568c9c8-98zs2 Liveness probe error: Get "https://10.217.0.57:8081/live": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)... 20m Warning Unhealthy pod/logging-loki-index-gateway-0 Readiness probe failed: Get "https://10.217.0.61:3101/ready": net/http: request canceled (Client.Timeout exceeded while awaiting headers) 20m Warning ProbeError pod/logging-loki-index-gateway-0 Readiness probe error: Get "https://10.217.0.61:3101/ready": net/http: request canceled (Client.Timeout exceeded while awaiting headers)... 20m Warning ProbeError pod/logging-loki-querier-6dcbdf8bb8-rbp8f Liveness probe error: Get "https://10.217.0.54:3101/loki/api/v1/status/buildinfo": net/http: request canceled (Client.Timeout exceeded while awaiting headers)... 20m Warning Unhealthy pod/logging-loki-querier-6dcbdf8bb8-rbp8f Liveness probe failed: Get "https://10.217.0.54:3101/loki/api/v1/status/buildinfo": net/http: request canceled (Client.Timeout exceeded while awaiting headers) 20m Warning Unhealthy pod/logging-loki-gateway-54c568c9c8-mwlnd Liveness probe failed: Get "https://10.217.0.56:8081/live": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) 20m Warning ProbeError pod/logging-loki-gateway-54c568c9c8-mwlnd Liveness probe error: Get "https://10.217.0.56:8081/live": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)... 20m Warning Unhealthy pod/logging-loki-gateway-54c568c9c8-mwlnd Liveness probe failed: Get "https://10.217.0.56:8083/live": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) 20m Warning ProbeError pod/logging-loki-gateway-54c568c9c8-mwlnd Liveness probe error: Get "https://10.217.0.56:8083/live": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)... 20m Warning Unhealthy pod/logging-loki-query-frontend-ff66c4dc9-5d4kx Liveness probe failed: Get "https://10.217.0.55:3101/loki/api/v1/status/buildinfo": net/http: request canceled (Client.Timeout exceeded while awaiting headers) 20m Warning Unhealthy pod/logging-loki-ingester-0 Readiness probe failed: Get "https://10.217.0.58:3101/ready": net/http: request canceled (Client.Timeout exceeded while awaiting headers) 20m Warning Unhealthy pod/logging-loki-gateway-54c568c9c8-98zs2 Liveness probe failed: Get "https://10.217.0.57:8083/live": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) 20m Warning ProbeError pod/logging-loki-compactor-0 Readiness probe error: Get "https://10.217.0.60:3101/ready": net/http: request canceled (Client.Timeout exceeded while awaiting headers)... 20m Warning Unhealthy pod/logging-loki-compactor-0 Readiness probe failed: Get "https://10.217.0.60:3101/ready": net/http: request canceled (Client.Timeout exceeded while awaiting headers) 20m Warning ProbeError pod/logging-loki-ingester-0 Readiness probe error: Get "https://10.217.0.58:3101/ready": net/http: request canceled (Client.Timeout exceeded while awaiting headers)... 20m Warning ProbeError pod/logging-loki-gateway-54c568c9c8-98zs2 Liveness probe error: Get "https://10.217.0.57:8083/live": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)... 20m Warning Unhealthy pod/logging-loki-gateway-54c568c9c8-98zs2 Liveness probe failed: Get "https://10.217.0.57:8081/live": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) 20m Warning ProbeError pod/logging-loki-query-frontend-ff66c4dc9-5d4kx Liveness probe error: Get "https://10.217.0.55:3101/loki/api/v1/status/buildinfo": net/http: request canceled (Client.Timeout exceeded while awaiting headers)... 20m Warning Unhealthy pod/logging-loki-index-gateway-0 Liveness probe failed: Get "https://10.217.0.61:3101/loki/api/v1/status/buildinfo": net/http: request canceled (Client.Timeout exceeded while awaiting headers) 20m Warning Unhealthy pod/logging-loki-ingester-0 Liveness probe failed: Get "https://10.217.0.58:3101/loki/api/v1/status/buildinfo": net/http: request canceled (Client.Timeout exceeded while awaiting headers) 20m Warning ProbeError pod/logging-loki-compactor-0 Liveness probe error: Get "https://10.217.0.60:3101/loki/api/v1/status/buildinfo": net/http: request canceled (Client.Timeout exceeded while awaiting headers)... 20m Warning ProbeError pod/logging-loki-ingester-0 Liveness probe error: Get "https://10.217.0.58:3101/loki/api/v1/status/buildinfo": net/http: request canceled (Client.Timeout exceeded while awaiting headers)... 20m Warning Unhealthy pod/logging-loki-compactor-0 Liveness probe failed: Get "https://10.217.0.60:3101/loki/api/v1/status/buildinfo": net/http: request canceled (Client.Timeout exceeded while awaiting headers) 20m Warning ProbeError pod/logging-loki-index-gateway-0 Liveness probe error: Get "https://10.217.0.61:3101/loki/api/v1/status/buildinfo": net/http: request canceled (Client.Timeout exceeded while awaiting headers)... 20m Warning Unhealthy pod/logging-loki-gateway-54c568c9c8-mwlnd Readiness probe failed: Get "https://10.217.0.56:8083/ready": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) 20m Warning ProbeError pod/logging-loki-gateway-54c568c9c8-98zs2 Readiness probe error: Get "https://10.217.0.57:8083/ready": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)... 20m Warning Unhealthy pod/logging-loki-gateway-54c568c9c8-98zs2 Readiness probe failed: Get "https://10.217.0.57:8083/ready": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) 20m Warning ProbeError pod/logging-loki-gateway-54c568c9c8-98zs2 Readiness probe error: Get "https://10.217.0.57:8081/ready": net/http: request canceled (Client.Timeout exceeded while awaiting headers)... 20m Warning Unhealthy pod/logging-loki-gateway-54c568c9c8-98zs2 Readiness probe failed: Get "https://10.217.0.57:8081/ready": net/http: request canceled (Client.Timeout exceeded while awaiting headers) 20m Warning Unhealthy pod/logging-loki-gateway-54c568c9c8-mwlnd Readiness probe failed: Get "https://10.217.0.56:8081/ready": net/http: request canceled (Client.Timeout exceeded while awaiting headers) 20m Warning ProbeError pod/logging-loki-gateway-54c568c9c8-mwlnd Readiness probe error: Get "https://10.217.0.56:8081/ready": net/http: request canceled (Client.Timeout exceeded while awaiting headers)... 20m Warning ProbeError pod/logging-loki-gateway-54c568c9c8-mwlnd Readiness probe error: Get "https://10.217.0.56:8083/ready": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)... 20m Warning Unhealthy pod/logging-loki-distributor-9c6b6d984-h7xw6 Readiness probe failed: Get "https://10.217.0.53:3101/ready": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) 20m Warning ProbeError pod/logging-loki-distributor-9c6b6d984-h7xw6 Readiness probe error: Get "https://10.217.0.53:3101/ready": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)... 20m Warning Unhealthy pod/logging-loki-gateway-54c568c9c8-mwlnd Readiness probe failed: Get "https://10.217.0.56:8083/ready": net/http: request canceled (Client.Timeout exceeded while awaiting headers) 20m Warning ProbeError pod/logging-loki-gateway-54c568c9c8-mwlnd Readiness probe error: Get "https://10.217.0.56:8083/ready": net/http: request canceled (Client.Timeout exceeded while awaiting headers)... 20m Warning ProbeError pod/logging-loki-querier-6dcbdf8bb8-rbp8f Readiness probe error: Get "https://10.217.0.54:3101/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 20m Warning Unhealthy pod/logging-loki-querier-6dcbdf8bb8-rbp8f Readiness probe failed: Get "https://10.217.0.54:3101/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 20m Warning ProbeError pod/logging-loki-query-frontend-ff66c4dc9-5d4kx Readiness probe error: Get "https://10.217.0.55:3101/loki/api/v1/status/buildinfo": net/http: request canceled (Client.Timeout exceeded while awaiting headers)... 20m Warning Unhealthy pod/logging-loki-query-frontend-ff66c4dc9-5d4kx Readiness probe failed: Get "https://10.217.0.55:3101/loki/api/v1/status/buildinfo": net/http: request canceled (Client.Timeout exceeded while awaiting headers) 20m Warning Unhealthy pod/logging-loki-gateway-54c568c9c8-98zs2 Readiness probe failed: Get "https://10.217.0.57:8083/ready": net/http: request canceled (Client.Timeout exceeded while awaiting headers) 20m Warning ProbeError pod/logging-loki-gateway-54c568c9c8-98zs2 Readiness probe error: Get "https://10.217.0.57:8083/ready": net/http: request canceled (Client.Timeout exceeded while awaiting headers)... 20m Warning Unhealthy pod/logging-loki-distributor-9c6b6d984-h7xw6 Readiness probe failed: Get "https://10.217.0.53:3101/ready": net/http: request canceled (Client.Timeout exceeded while awaiting headers) 20m Warning ProbeError pod/logging-loki-distributor-9c6b6d984-h7xw6 Readiness probe error: Get "https://10.217.0.53:3101/ready": net/http: request canceled (Client.Timeout exceeded while awaiting headers)...